- 3099 Views
- 5 replies
- 2 kudos
Resolved! Cluster by auto pyspark
I can find documentation to enable automatic liquid clustering with SQL code: CLUSTER BY AUTO. But how do I do this with Pyspark? I know I can do it with spark.sql("ALTER TABLE CLUSTER BY AUTO") but ideally I want to pass it as an .option().Thanks in...
- 3099 Views
- 5 replies
- 2 kudos
- 2 kudos
This is supported now for DBR 16.4+ for both DataframeWriterV1 and DataframeWriterV2 APIs, and also for DLT, and DataStreaming APIs. More details are here: https://docs.databricks.com/aws/en/delta/clustering . Basically using the option, `.option("cl...
- 2 kudos
- 332 Views
- 3 replies
- 3 kudos
where do I find the AI/BI dashboard?
I only see dashboard under the header SQL on the left menu, but I expected a header data. Got the feeling that under SQL, the dashboard lacks functionality.Best regards,Stefaan
- 332 Views
- 3 replies
- 3 kudos
- 3 kudos
@Neirynck I'd HIGHLY recommend checking out Databricks' Youtube channel or Demo Center when you're exploring part of the product. I'll link them below. @Neirynck as for your question, here's a video on AI/BI that actually sold Databricks to me: https...
- 3 kudos
- 11176 Views
- 16 replies
- 6 kudos
Unable to Login - Account Verification Loop
I'm having trouble logging in to my Databricks account at databricks.com. Here's what happens:I enter my email address and password.I receive an account verification code via email.I enter the verification code on the login page.Instead of logging me...
- 11176 Views
- 16 replies
- 6 kudos
- 6 kudos
Same problem here for me I tried 10 different resend codes and nothing works. It keeps telling me invalid code.
- 6 kudos
- 123 Views
- 0 replies
- 1 kudos
WHERE WE SHOULD HOST THE NEXT MEETUP IN THE US?
Hi all, I collaborate with a Databricks partner, and we've started organizing in-person Meetups in the states. We want to arrange another one, possibly in December or late January. Where do you think most people will be interested in joining? It's fr...
- 123 Views
- 0 replies
- 1 kudos
- 2621 Views
- 1 replies
- 0 kudos
Databricks workspace
I have one issue . I have created delta table and vector search index from the delta table. For a particular query if I am doing similarity search then sometimes I am getting the documents and sometimes I am not getting any documents. For example# qu...
- 2621 Views
- 1 replies
- 0 kudos
- 0 kudos
This inconsistent behavior in your Delta Table and vector search index is a common issue with semantic vector searches, especially when working with diverse or structured data like Excel file contents. There are several likely causes for why your sim...
- 0 kudos
- 302 Views
- 1 replies
- 1 kudos
Resolved! What’s the difference between the Free/Community Edition vs the fully paid version of Databricks, an
Databricks has a free version (called Community Edition) and a paid version. What are the main differences between them, and what things can’t I do in the free version that I can do in the paid one?
- 302 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Suheb .The Free Edition is intended for students, hobbyists, and aspiring data and AI professionals. It is not intended for commercial use. In addition, the Free Edition is subject to the following limitations:Databricks Free Edition limitations ...
- 1 kudos
- 186 Views
- 1 replies
- 0 kudos
Missing warehouse id/metadata for the system compute warehouse table
I ran the following queries for a specific warehouse_id = '54a93d2138433216' SELECT * FROM system.billing.usage WHERE usage_metadata.warehouse_id = '54a93d2138433216';SELECT * FROM system.compute.warehouse_events WHERE warehouse_id = '54a93d213843321...
- 186 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Danish1105! This post duplicates a recent post you shared. A response has already been provided there. Please continue the discussion in that thread to keep the conversation organized.
- 0 kudos
- 357 Views
- 4 replies
- 0 kudos
Resolved! What’s the easiest way to import my local dataset into Databricks for analysis?
How can I upload a data file (like a CSV, Excel, or JSON file) from my computer into Databricks so I can start analyzing it there?
- 357 Views
- 4 replies
- 0 kudos
- 0 kudos
For large size relational Databases like from Oracle how can we move to Azure databricks. Non-unity catalog to unity catalog?
- 0 kudos
- 390 Views
- 2 replies
- 1 kudos
Resolved! How does Databricks handle versioning of notebooks or jobs, and what good practices should newcomers
When you create notebooks or jobs in Databricks, how does Databricks keep track of different versions or changes? And what should beginners do to manage versions safely and effectively?
- 390 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Suheb,That’s a great question, version control is one of the most important things to get right early on.As a best practice, you should never run notebooks directly in production. Instead, notebooks should be treated as development assets, once v...
- 1 kudos
- 290 Views
- 1 replies
- 1 kudos
Python Async db connection to databricks
How to connect to azure databricks using sqlalchemy asynchronously
- 290 Views
- 1 replies
- 1 kudos
- 1 kudos
Databricks SQL Connector is fundamentally synchronous - it doesn't have native async/await support. This creates an inherent mismatch when you want to use it in an async Python application. Why This Matters When you make a database query ...
- 1 kudos
- 318 Views
- 3 replies
- 3 kudos
Streamlining App Development with Databricks Marketplace and API Integrations
Hi everyone,I’m interested in discussing how Databricks Marketplace and API integrations can be leveraged to speed up and simplify application development. With the growing need for data-driven apps, I’d like to hear from the community about practica...
- 318 Views
- 3 replies
- 3 kudos
- 3 kudos
Leveraging Databricks Marketplace and APIs can significantly speed up data-driven app development. The Marketplace gives quick access to validated datasets, ML models, and connectors, reducing time spent on sourcing and infrastructure setup. For inte...
- 3 kudos
- 485 Views
- 1 replies
- 1 kudos
Creating links in embedded dashboards
I am curious what Databricks AI/BI's capabilities are in terms of linking out other dashboards in your dashboards. Can I create a text box that links out another dashboard? I noticed you can also create links in a pivot table. What will happen if I e...
- 485 Views
- 1 replies
- 1 kudos
- 1 kudos
You can add a text widget to the canvas to insert hyperlinks to any URL, including other dashboards.In pivot tables, you can define URL paths for cell values from the Actions tab to make them clickable.For navigation within a dashboard, use “drill th...
- 1 kudos
- 222 Views
- 1 replies
- 0 kudos
Attempting to Use VSCode Extension: Error: No sync destination found
Hello,I have the Databricks Extension + Databricks Connect Python module installed.I am connected to the target environment with a running cluster and I have all green checkmarks under the "Configuration" section.However, when testing a python file a...
- 222 Views
- 1 replies
- 0 kudos
- 0 kudos
The error message "No sync destination found" when running your Python file in Databricks with the Extension and Databricks Connect module, even with all green configuration checkmarks, typically indicates a configuration issue with the sync destinat...
- 0 kudos
- 211 Views
- 2 replies
- 0 kudos
What’s the easiest way to clean and transform data using PySpark in Databricks?
You have some raw data (like messy Excel files, CSVs, or logs) and you want to prepare it for analysis — by removing errors, fixing missing values, changing formats, or combining columns — using PySpark (Python for Apache Spark) inside Databricks.
- 211 Views
- 2 replies
- 0 kudos
- 0 kudos
The easiest way to clean and transform data using PySpark in Databricks is by leveraging the DataFrame API. Start by loading data into a Spark DataFrame with spark.read. Use built-in functions like dropna, fillna, and withColumn to handle missing val...
- 0 kudos
- 309 Views
- 1 replies
- 0 kudos
How to Leverage Databricks for End-to-End AI Model Development
Hi everyone,I’m exploring how to use Databricks as a platform for end-to-end AI and machine learning model development, and I’d love to get insights from professionals and practitioners who have hands-on experience.Specifically, I’m curious about:Set...
- 309 Views
- 1 replies
- 0 kudos
- 0 kudos
You can leverage Databricks for end-to-end AI model development by using its Lakehouse Platform, which unifies data engineering, analytics, and machine learning in one workspace. Start by ingesting and transforming data using Apache Spark and Delta L...
- 0 kudos
