-
-
Notifications
You must be signed in to change notification settings - Fork 638
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add RegExp
support to PoolCluster
#3451
feat: add RegExp
support to PoolCluster
#3451
Conversation
Co-authored-by: Weslley Araújo <[email protected]>
Thanks, @uPaymeiFixit 🙋🏻♂️ Do you think it would be possible to include a failing test without your changes? I would try an alternative similar to the original (without using |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3451 +/- ##
==========================================
+ Coverage 88.97% 89.08% +0.11%
==========================================
Files 86 86
Lines 13527 13531 +4
Branches 1564 1569 +5
==========================================
+ Hits 12035 12054 +19
+ Misses 1492 1477 -15
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
RegExp
support to PoolCluster
About the lint errors, you can run |
@wellwelwel I'm not sure if I understand what you mean by a failing test without my changes. Maybe a useful test would be checking that if you run |
This PR includes both a feature (support for It would be great to include a test covering the import mysql from "mysql2/promise";
(async () => {
const poolCluster = mysql.createPoolCluster();
poolCluster.add("129", {
user: "root",
database: "exampledb",
});
poolCluster.add("12", {
user: "root",
database: "exampledb",
});
let connection = await poolCluster.getConnection("125");
// connection may be ID 129 or ID 12. Should be POOL_NOEXIST error.
await connection.query("SELECT 1");
poolCluster.end();
})();
That's a good idea 🙋🏻♂️ EDIT: In #3261, we created a new test for Pool Cluster. Just rebasing to trigger it here. |
@wellwelwel I think I've got the fix-related test down, but I'm struggling a bit with the RegEx test. I've created a new file, 'use strict';
const { assert } = require('poku');
const common = require('../../common.test.cjs');
const process = require('node:process');
// TODO: config poolCluster to work with MYSQL_CONNECTION_URL run
if (`${process.env.MYSQL_CONNECTION_URL}`.includes('pscale_pw_')) {
console.log('skipping test for planetscale');
process.exit(0);
}
const cluster = common.createPoolCluster();
const poolConfig = common.getConfig();
cluster.add('SLAVE1', poolConfig);
cluster.of(/SLAVE[12]/).getConnection((error, connection) => {
assert.equal(connection._clusterId, 'SLAVE1', 'should match regex');
});
cluster.end(); And it's throwing this error:
Any ideas? |
This error is due to the callback not waiting for execution to end the connection. You can end the connection before starting the assertions or use a promise-based connection 🙋🏻♂️ |
I've added what I think are valid tests. I'm not familiar with poku or the folder structure for these tests, so it's entirely possible they're in the wrong place. Please let me know if they could be better organized. |
Poku API follows a similar usage as it is for Node.js (poku.io/docs/documentation/assert), for example:
import assert from 'node:assert';
import test from 'node:test';
test('My Test', () => {
assert(true);
assert.ok(true);
});
// In Node.js, asynchronous tests do not follow the standard JavaScript flow/behavior
test('My Test', async () => {
assert(true);
assert.ok(true);
});
import { test, assert } from 'poku';
test('My Test', () => {
assert(true);
assert.ok(true);
});
// In Poku, asynchronous tests follow the standard JavaScript flow/behavior
await test('My Test', async () => {
assert(true);
assert.ok(true);
}); |
I've seen that const source = pattern
.replace(/([.+?^=!:${}()|[\]/\\])/g, '\\$1')
.replace(/\*/g, '.*');
return new RegExp(`^${source}$`); But I'd like to test first if it's possible to get the same result without building |
Off the top of my head I can't think of a clean way to achieve the same result without building the regular expression manually. This comes to mind, but it's not fully compatible with mysqljs/mysql. Are you more worried about performance, or clarity? --- lib/pool_cluster.js
+++ lib/pool_cluster.js
@@ -269,7 +269,10 @@ class PoolCluster extends EventEmitter {
let foundNodeIds = this._findCaches[pattern];
if (foundNodeIds === undefined) {
- const expression = patternRegExp(pattern);
+ let expression = pattern;
+ if (typeof pattern === 'string' && pattern.at(-1) === '*') {
+ expression = patternRegExp(pattern);
+ }
foundNodeIds = this._serviceableNodeIds.filter((id) =>
id.match(expression) |
Performance is a consideration, but my concern is particularly about security against ReDoS. In this project you can see how I usually handle recursions to create safe dynamics "matches" in a similar result of a dynamic But it's not a requirement, it's more of a curiosity. I'll try something and I'll come back with some idea or make sure there's no ReDoS possibility. |
I tried something and, performance-wise, I suppose it's better, but complexity without using _findNodeIds(pattern, includeOffline) {
let currentTime = 0;
let foundNodeIds = this._findCaches[pattern];
if (foundNodeIds !== undefined) {
if (includeOffline) {
return foundNodeIds;
}
return foundNodeIds.filter((nodeId) => {
const node = this._getNode(nodeId);
if (!node._offlineUntil) return true;
currentTime = currentTime || getMonotonicMilliseconds();
return node._offlineUntil <= currentTime;
});
}
if (pattern instanceof RegExp) {
// RegExp
foundNodeIds = this._serviceableNodeIds.filter((id) => pattern.test(id));
} else if (pattern === '*') {
// all
foundNodeIds = this._serviceableNodeIds;
} else if (this._serviceableNodeIds.indexOf(pattern) !== -1) {
// one
foundNodeIds = [pattern];
} else if (pattern.indexOf('*') !== -1) {
// wild matching
const parts = pattern.split('*');
foundNodeIds = this._serviceableNodeIds.filter((id) => {
let idToCheck = id;
if (parts[0] && !idToCheck.startsWith(parts[0])) {
return false;
}
idToCheck = idToCheck.slice(parts[0].length);
for (let i = 1; i < parts.length - 1; i++) {
const part = parts[i];
if (part === '') continue;
const index = idToCheck.indexOf(part);
if (index === -1) return false;
idToCheck = idToCheck.slice(index + part.length);
}
const lastPart = parts[parts.length - 1];
if (lastPart && !idToCheck.endsWith(lastPart)) {
return false;
}
return true;
});
} else {
foundNodeIds = [];
}
this._findCaches[pattern] = foundNodeIds;
if (includeOffline) {
return foundNodeIds;
}
return foundNodeIds.filter((nodeId) => {
const node = this._getNode(nodeId);
if (!node._offlineUntil) {
return true;
}
if (!currentTime) {
currentTime = getMonotonicMilliseconds();
}
return node._offlineUntil <= currentTime;
});
} On the other hand, the dynamic @uPaymeiFixit and @sidorares, please let me know what do you think 🙋🏻♂️ |
I think the only way ReDoS is possible is if the user provides an instanceof RegExp as the pattern (because if they provide a string all of the regex special characters are escaped), in which case in my opinion it's up to them to sanitize the expression. I can appreciate the effort that went into experimenting with that alternative, but I agree it feels a little complex. Maybe it could be a little easier if some of that logic gets broken out into another function, but I think my vote is still dynamic regular expression. |
Coming back, the concern about security remains. The problem is not the use of My concern is with the process of dynamically creating strings into regular expressions, especially considering users rely on this value being assigned to a string and have never been introduced to pre-validating these value as a final
pattern
.replace(/([.+?^=!:${}()|[\]/\\])/g, '\\$1')
.replace(/\*/g, '.*')
I understand it's unusual, but here are two RCE vulnerabilities fixed with severity |
If that's the case, I agree. Because it certainly has me fooled. Just to help me understand, can give an example of a string which would not be properly sanitized by |
Just found the answer in the docs: So in that case it makes sense not to normalize Since this behavior is expected and documented in mysqljs/mysql, I believe that this matter can be considered resolved. Thank you for your patience, @uPaymeiFixit 🤝 |
mysqljs/mysql has been updated to support regular expressions in PoolCluster. This brings most of that code to mysql2.
Closes #3450
Closes #503