NPMREC: NPM Packages and Similar Projects Recommendation System

Proceedings of the 15th International Conference on Mining Software Repositories

As the popularity of the JavaScript language is constantly increasing, one of the most important challenges today is to assess the quality of JavaScript packages. Developers often employ tools for code linting and for the extraction of static analysis metrics in order to assess and/or improve their code. In this context, we have developed npn-miner, a platform that crawls the npm registry and analyzes the packages using static analysis tools in order to extract detailed quality metrics as well as high-level quality attributes, such as maintainability and security. Our infrastructure includes an index that is accessible through a web interface, while we have also constructed a dataset with the results of a detailed analysis for 2000 popular npm packages.

IEEE Transactions on Software Engineering, 2022

Nowadays, developing software would be unthinkable without the use of third-party packages. Although such code reuse helps to achieve rapid continuous delivery of software to end-users, blindly reusing code has its pitfalls. For example, prior work has investigated the rationale for using packages that implement simple functionalities, known as trivial packages. This prior work showed that although these trivial packages were simple, they were popular and prevalent in the npm ecosystem. This popularity and prevalence of trivial packages peaked our interest in questioning the 'triviality of trivial packages'. To better understand and examine the triviality of trivial packages, we mine a large set of JavaScript projects that use trivial npm packages and evaluate their relative centrality. Specifically, we evaluate the triviality from two complementary points of view: based on project usage and ecosystem usage of these trivial packages. Our result shows that trivial packages are being used in central JavaScript files of a software project. Additionally, by analyzing all external package API calls in these JavaScript files, we found that a high percentage of these API calls are attributed to trivial packages. Therefore, these packages play a significant role in JavaScript files. Furthermore, in the package dependency network, we observed that 16.8% packages are trivial and in some cases removing a trivial package can impact approximately 29% of the ecosystem. Overall, our finding indicates that although smaller in size and complexity, trivial packages are highly depended on packages by JavaScript projects. Additionally, our study shows that although they might be called trivial, nothing about trivial packages is trivial.

2021

Code reuse is an important part of software development. The adoption of code reuse practices is especially noticeable in Node.js. The Node.js package manager, NPM, contains over 1 Million packages and developers often seek out packages to solve programming tasks. Due to the vast number of packages, selecting the right package is difficult and time consuming. With the goal of improving productivity of developers that heavily reuse code through third-party packages, we present Node Code Query (NCQ), a custom Read-Eval-Print Loop environment that allows developers to 1) search for NPM packages using natural language queries, 2) search for code snippets related to those packages, 3) quickly setup new environments for testing those snippets, and 4) transition between search and editing modes. Our user study shows that participants began programming faster and concluded tasks faster with NCQ than with a baseline approach, and that they liked, among other features, the search for code sni...

Virginia Tech, 2020

The meteoric increase in the popularity of JavaScript and a large developer community has led to the emergence of a large ecosystem of third-party packages available via the Node Package Manager (NPM) repository which contains over one million published packages and witnesses a billion daily downloads. Most of the developers download these pre-compiled published packages from the NPM repository instead of building these packages from the available source code. Unfortunately, recent articles have revealed repackaging attacks to the NPM packages. To achieve such attacks the attackers primarily follow three steps – (1) download the source code of a highly depended upon NPM package, (2) inject malicious code, and (3) then publish the modified packages as either misnamed package (i.e., typo-squatting attack) or as the official package on the NPM repository using compromised maintainer credentials. These attacks highlight the need to verify the reproducibility of NPM packages. Reproducible Build is a concept that allows the verification of build artifacts for pre-compiled packages by re-building the packages using the same build environment configuration documented by the package maintainers. This motivates us to conduct an empirical study (1) to examine the reproducibility of NPM packages, (2) to assess the influence of any non-reproducible packages, and (3) to explore the reasons for non-reproducibility. Firstly, we downloaded all versions/releases of 226 most-depended upon NPM packages, and then built each version with the available source code on Github. Secondly, we applied diffoscope, a differencing tool to compare the versions we built against the version downloaded from the NPM repository. Finally, we did a systematic investigation of the reported differences. At least one version of 65 packages was found to be non-reproducible. Moreover, these non- reproducible packages have been downloaded millions of times per week which could impact a large number of users. Based on our manual inspection and static analysis, most reported differences were semantically equivalent but syntactically different. Such differences result due to non-deterministic factors in the build process. Also, we infer that semantic differences are introduced because of the shortcomings in the JavaScript uglifiers. Our research reveals challenges of verifying the reproducibility of NPM packages with existing tools, reveal the point of failures using case studies, and sheds light on future directions to develop better verification tools.

arXiv (Cornell University), 2022

The selection of software technologies is an important but complex task. We consider developers of JavaScript (JS) applications, for whom the assessment of JS libraries has become difficult and time-consuming due to the growing number of technology options available. A common strategy is to browse software repositories via search engines (e.g., NPM, or Google), although it brings some problems. First, given a technology need, the engines might return a long list of results, which often causes information overload issues. Second, the results should be ranked according to criteria of interest for the developer. However, deciding how to weight these criteria to make a decision is not straightforward. In this work, we propose a two-phase approach for assisting developers to retrieve and rank JS technologies in a semi-automated fashion. The first-phase (ST-Retrieval) uses a meta-search technique for collecting JS technologies that meet the developer's needs. The second-phase (called ST-Rank), relies on a machine learning technique to infer, based on criteria used by other projects in the Web, a ranking of the output of ST-Retrieval. We evaluated our approach with NPM and obtained satisfactory results in terms of the accuracy of the technologies retrieved and the order in which they were ranked.

IEEE T-IFS

The Node Package Manager (NPM) registry contains millions of JavaScript packages widely shared between worldwide developers. However, NPM has also been abused by attackers to spread malicious packages, highlighting the importance of detecting malicious NPM packages. Existing malicious NPM package detectors suffer from, among other things, high false positives and/or high false negatives. In this paper, we propose a novel Malicious NPM Package Detector (MalPacDetector), which leverages Large Language Model (LLM) to automatically and dynamically generate features (rather than asking experts to manually define them). To evaluate the effectiveness of Mal-PacDetector and existing detectors, we construct a new NPM package dataset, which overcomes the weaknesses of existing datasets (e.g., a small number of examples and a high repetition rate of malicious fragments). The experimental results show that MalPacDetector outperforms existing detectors by achieving a false positive rate of 1.3% and a false negative rate of 7.5%. In particular, MalPacDetector detects 39 previously unknown malicious packages, which are confirmed by the NPM security team.

This work presents findings and mitigations on an understudied issue, which we term shrinkwrapped clones, that is endemic to the npm software package ecosystem. A shrinkwrapped clone is a package which duplicates, or near-duplicates, the code of another package without any indication or reference to the original package. This phenomenon represents a challenge to the hygiene of package ecosystems, as a clone package may siphon interest from the package being cloned, or create hidden duplicates of vulnerable, insecure code which can fly under the radar of audit processes. Motivated by these considerations, we propose unwrapper, a mechanism to programmatically detect shrinkwrapped clones and match them to their source package. unwrapper uses a package difference metric based on directory tree similarity, augmented with a prefilter which quickly weeds out packages unlikely to be clones of a target. Overall, our prototype can compare a given package within the entire npm ecosystem (1,716,061 packages with 20,190,452 different versions) in 72.85 seconds, and it is thus practical for live deployment. Using our tool, we performed an analysis of a subset of npm packages, which resulted in finding up to 6,292 previously unknown shrinkwrapped clones, of which up to 207 carried vulnerabilities from the original package that had already been fixed in the original package. None of such vulnerabilities were discoverable via the standard npm audit process.

arXiv (Cornell University), 2022

Many startups and companies worldwide have been using project management software and tools to monitor, track and manage their projects. For software projects, the number of tasks from the beginning to the end is quite a large number that sometimes takes a lot of time and effort to search and link the current task to a group of previous ones for further references. This paper proposes an efficient task dependency recommendation algorithm to suggest tasks dependent on a given task that the user has just created. We present an efficient feature engineering step and construct a deep neural network to this aim. We performed extensive experiments on two different large projects (MDLSITE from moodle.org and FLUME from apache.org) to find the best features in 28 combinations of features and the best performance model in the combination of two embedding methods (GloVe and FastText). We consider three types of models (GRU, CNN, LSTM) using Accuracy@K, MRR@K, and Recall@K (where K = 1, 2, 3, and 5) and baseline models using traditional methods: TF-IDF with various matching score calculating such as cosine similarity, Euclidean distance, Manhattan distance, and Chebyshev distance. After many experiments, GloVe Embedding and CNN model reached the best result in our dataset so we decided to choose this model as our

International Journal of Advanced Trends in Computer Science and Engineering , 2021

GitHub is a product improvement stage that advances participation and joint exertion in project advancement. Generally, engineers investigate related activities to reuse capacities and during investigating they coincidentally find a few elements that might help them in their undertaking. Recommending developers some projects that are similar to their work can save their time. Yet, it is difficult getting relevant projects amongst the pool of many projects on GitHub. Besides, every other user may have different requirements and choices for the project. A recommendation system saves developers from spending their time searching for projects that can help them in developing. In this paper, we propose an interactive and customized recommendation approach that recognizes software project features and developer behavior. This proposed approach naturally suggests the top-N most comparable programming projects. The outcomes utilizing information slithered from GitHub shows that our proposed approach suggest significant tasks with high review at cutoff 5 and 10.

npm ecosystem is crucial for the JavaScript community and its development is significantly influenced by the opinions and feedback of core software developers.Many software developers have utilized social media, such as Twitter, to share community-related information and their views. However, the communication between npm developers via Twitter in terms of topics, nature, and sentiment have not been analyzed.This study conducts an empirical analysis of tweets by core software developers related to the npm ecosystem to understand their perceptions and opinions better. A dataset of tweets was collected and analyzed using qualitative analysis techniques to identify the topic of tweets, their nature and their sentiments. Our study demonstrated that most tweets belong to the package management category followed by notifications and community-related information.The most frequently discussed topics among core software developers in the package management category are usage scenarios. It a...