DATAMIMIC Community Edition

🧠 AI-powered test data generation platform enabling developers to create realistic, scalable, and privacy-compliant test data. Features model-driven data generation, GDPR compliance, and seamless P…

Productivity

faker

data-simulation

data-privacy

data-generation

DATAMIMIC Community Edition

Introduction

Welcome to DATAMIMIC Community Edition, the AI-powered platform that revolutionizes test data generation! By leveraging advanced AI and model-driven technologies, DATAMIMIC enables developers and testers to create realistic, scalable, and privacy-compliant test data with ease.

Key Features

🧠 Model-Driven Data Generation: Utilize sophisticated algorithms to simulate real-world data scenarios.
🔮 AI-Powered Data Generation: Simulate real-world data scenarios using cutting-edge AI algorithms. (Like GANs, LLMs, and more)
🛡️ Data Privacy Compliance: Anonymize and pseudonymize data to meet GDPR and global data protection standards.
🚀 High Performance: Engineered for scalability to handle complex datasets efficiently.
🐍 Seamless Python Integration: Easily integrate with Python projects and manage dependencies.
⚙️ Extensibility: Customize and extend functionalities to suit your specific testing needs.

Note: The Community Edition focuses on core functionalities and does not include AI-powered features like automatic model generation. These advanced features are available in the Enterprise Edition.

Why Use DATAMIMIC?

Traditional test data generation can be time-consuming and may compromise data privacy. DATAMIMIC addresses these challenges by:

Reducing Time-to-Market: Quickly generate test data without manual intervention.
Enhancing Test Coverage: Simulate diverse data scenarios for comprehensive testing.
Ensuring Compliance: Maintain data privacy and comply with legal regulations.
Improving Data Quality: Generate realistic data that mirrors production environments.

Installation

Prerequisites

Operating System: Windows, macOS, or Linux
Python: Version 3.10 or higher
Optional: uv Package Manager for development setup GitHub

User Installation

The simplest way to get started with DATAMIMIC is through pip:

pip install datamimic-ce

Verify the installation:

datamimic --version

Developer Installation

For contributors and developers who want to work with the source code:

Install uv Package Manager:
```
pip install uv
```

Clone and set up the repository:

git clone https://github.com/rapiddweller/datamimic.git
cd datamimic
uv sync

Usage Guide

Basic Usage

Create a new data generation project:

datamimic init my-project
cd my-project

Configure your data model in datamimic.xml:

<setup>
    <generate name="datamimic_user_list" count="100" target="CSV,JSON">
        <variable name="person" entity="Person(min_age=18, max_age=90, female_quota=0.5)"/>
        <key name="id" generator="IncrementGenerator"/>
        <key name="first_name" script="person.given_name"/>
        <key name="last_name" script="person.family_name"/>
        <key name="gender" script="person.gender"/>
        <key name="birthDate" script="person.birthdate" converter="DateFormat('%d.%m.%Y')"/>
        <key name="email" script="person.family_name + '@' + person.given_name + '.de'"/>
        <key name="ce_user" values="True, False"/>
        <key name="ee_user" values="True, False"/>
        <key name="datamimic_lover" constant="DEFINITELY"/>
    </generate>
</setup>

Generate data:
```
datamimic run datamimic.xml
```

Access the generated data in the output directory.

json export:

[
{"id": 1, "first_name": "Mary", "last_name": "Mcgowan", "gender": "female", "birthDate": "1946-05-15T00:00:00", "email": "Mcgowan@Mary.de", "ce_user": false, "ee_user": true, "datamimic_lover": "DEFINITELY"},
{"id": 2, "first_name": "Gabrielle", "last_name": "Malone", "gender": "female", "birthDate": "1989-11-27T00:00:00", "email": "Malone@Gabrielle.de", "ce_user": false, "ee_user": true, "datamimic_lover": "DEFINITELY"},
{"id": 4, "first_name": "Margaret", "last_name": "Torres", "gender": "female", "birthDate": "2006-07-13T00:00:00", "email": "Torres@Margaret.de", "ce_user": false, "ee_user": false, "datamimic_lover": "DEFINITELY"},
{"id": 5, "first_name": "Monica", "last_name": "Meyers", "gender": "female", "birthDate": "1983-07-22T00:00:00", "email": "Meyers@Monica.de", "ce_user": true, "ee_user": false, "datamimic_lover": "DEFINITELY"},
{"id": 6, "first_name": "Jason", "last_name": "Davis", "gender": "male", "birthDate": "1941-07-05T00:00:00", "email": "Davis@Jason.de", "ce_user": true, "ee_user": false, "datamimic_lover": "DEFINITELY"},
{"...":  "..."},
{"id": 100, "first_name": "Jared", "last_name": "Rivas", "gender": "male", "birthDate": "1975-03-16T00:00:00", "email": "Rivas@Jared.de", "ce_user": true, "ee_user": true, "datamimic_lover": "DEFINITELY"}
]

csv export:

id|first_name|last_name|gender|birthDate|email|ce_user|ee_user|datamimic_lover
1|Mary|Mcgowan|female|1946-05-15 00:00:00|Mcgowan@Mary.de|False|True|DEFINITELY
2|Gabrielle|Malone|female|1989-11-27 00:00:00|Malone@Gabrielle.de|False|True|DEFINITELY
3|Antonio|Davis|male|2005-05-12 00:00:00|Davis@Antonio.de|False|True|DEFINITELY
4|Margaret|Torres|female|2006-07-13 00:00:00|Torres@Margaret.de|False|False|DEFINITELY
5|Monica|Meyers|female|1983-07-22 00:00:00|Meyers@Monica.de|True|False|DEFINITELY
...
100|Jason|Davis|male|1941-07-05 00:00:00|Davis@Jason.de|True|False|DEFINITELY

Advanced Features

DATAMIMIC supports various advanced features:

Custom Generators: Create your own data generators
Data Relationships: Define complex relationships between entities
Import/Export Formats CE: Support for JSON, XML, CSV, RDBMS and MongoDB
Import/Export Formats EE: Kafka, EDI, XSD and more
Data Anonymization: Anonymize data to comply with privacy regulations
Data Validation: Define and enforce data validation rules
Scripting: Extend functionality using Python scripts
Database Integration: Connect to databases for seamless data generation
Model-Driven Generation: Utilize models to generate realistic data
Validation Rules: Define and enforce data validation rules
Scripting: Extend functionality using Python scripts

Examples and Demos

Explore our demo collection:

# List available demos
datamimic demo list

# Run a specific demo
datamimic demo create demo-model
datamimic run ./demo-model/datamimic.xml

Key demos include:

Basic entity generation
Complex relationships
Database integration
Custom generator creation
Privacy compliance examples

Find more examples in the datamimic_ce/demos directory.

Contributing

We ❤️ contributions! Here's how you can help:

Code Contributions: Submit pull requests for new features or bug fixes.
Documentation: Improve existing docs or help with translations.
Community Engagement: Join discussions and support other users.

Check out our Contribution Guidelines and Code of Conduct.

License

DATAMIMIC CE is now open source and licensed under MIT:

📄 Open Source License: Licensed under the MIT License
🆓 Free for Everyone: Use freely for both personal and commercial projects
💡 Key Permissions:
- Commercial use
- Modification
- Distribution
- Private use

For questions or support, contact us at info@rapiddweller.com.

Support

Need help or have questions? We're here for you!

Connect with Us

Stay updated and connect with our community!

🌐 Website: www.datamimic.io
🏢 Rapiddweller: www.rapiddweller.com
💼 LinkedIn: rapiddweller
🐦 Twitter: @rapiddweller

Acknowledgments

A big thank you to all our contributors! Your efforts make DATAMIMIC possible.

Don't forget to ⭐ star and 👀 watch this repository to stay updated!

Legal Notices

For detailed licensing information, please see the LICENSE file.