- 1573 Views
- 2 replies
- 0 kudos
Resolved! Urgency: How to do Data Migration task using Databricks Lakebridge tool ?
Dear community expert,I have completed two phases Analyzer & Converter of Databricks Lakebridge but stuck at migrating data from source to target using lakebridge. I have watched BrickBites Series on Lakebridge but did not find on how to migrate data...
- 1573 Views
- 2 replies
- 0 kudos
- 0 kudos
To migrate tables and views from Snowflake (source) to Databricks (target) using Lakebridge, you must export your data from Snowflake into a supported cloud storage (usually as Parquet files), then import these files into Databricks Delta tables. Lak...
- 0 kudos
- 10158 Views
- 8 replies
- 0 kudos
How we can send databricks log to Azure Application Insight ?
Hi All,I want to send databricks logs to azure application insight.Is there any way we can do it ??Any blog or doc will help me.
- 10158 Views
- 8 replies
- 0 kudos
- 0 kudos
Hello,I finally used he AppInsights agent from OpenTelemetry which is documented in the official Microsoft documentation here:https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-enable?tabs=javaBelow is an adaptation of this "Get ...
- 0 kudos
- 8898 Views
- 3 replies
- 3 kudos
When to Use and when Not to Use Liquid Clustering?
Hi everyone,I’m looking for some practical guidance and experiences around when to choose Liquid Clustering versus sticking with traditional partitioning + Z-ordering.From what I’ve gathered so far:For small tables (<10TB), Liquid Clustering gives s...
- 8898 Views
- 3 replies
- 3 kudos
- 3 kudos
Deciding between Liquid Clustering and traditional partitioning with Z-ordering depends on table size, query patterns, number of clustering columns, and file optimization needs. For tables under 10TB with queries consistently filtered on 1–2 columns,...
- 3 kudos
- 1692 Views
- 4 replies
- 0 kudos
Resolved! MERGE operation not performing data skipping with liquid clustering on key columns
Hi, I need some help understanding a performance issue.I have a table that reads approximately 800K records every 30 minutes in an incremental manner.Let’s say its primary key is:timestamp, x, y This table is overwritten every 30 minutes and serves ...
- 1692 Views
- 4 replies
- 0 kudos
- 0 kudos
MERGE is not a pure read plus filter operationEven though Liquid Clustering organizes your data by key ranges and writes min/max stats, the MERGE engine has to identify both matches and non-matches.That means the query planner must:Scan all candidate...
- 0 kudos
- 4209 Views
- 3 replies
- 1 kudos
Resolved! Schema updating with CI/CD development in SQL
Hi all,I am working to resolve how to build tables in a development workspace catalog and then easily migrate the code to a production catalog without manually altering the schema name. For those unaware, you can't have the same catalog names in deve...
- 4209 Views
- 3 replies
- 1 kudos
- 1 kudos
I will like to see an example from databricks that how will alembic and databricks works together, especially with schema evo. I think once schema got changed by schema evo, the alembic version not valid anymore. Wondering how to handle it properly. ...
- 1 kudos
- 5954 Views
- 5 replies
- 2 kudos
Reading from an S3 bucket using boto3 on serverless cluster
Hello All,I am trying to read a CSV file from my S3 bucket in a notebook running on serverless.I am using the two standard functions below, but I get a credentials error (Error reading CSV from S3: Unable to locate credentials).I don't have this issu...
- 5954 Views
- 5 replies
- 2 kudos
- 2 kudos
Boto3 with Access/Secret Key worked. I will try the Service Credentials. If Databricks Documentation is right, Instance Profiles with Serverless should work to establish Boto3 connection, but, unfortunately, setting up instance profiles on Serverles...
- 2 kudos
- 2112 Views
- 2 replies
- 0 kudos
Urgency: Getting Lakebridge installation failed in our organization environment (laptop)
Dear community expert,I’m reaching out for assistance with installing Databricks Lakebridge on my organization laptop. I have confirmed the stated prerequisites are installed: Java 22+, Python 3.11+, and the latest Databricks CLI, but the installer f...
- 2112 Views
- 2 replies
- 0 kudos
- 0 kudos
Additionally, you can check the following documentation and links that I found, which may help you:1) To view the available Python versions and guidance on using a virtual environment (virtualenv or conda):https://www.piwheels.org/project/databricks-...
- 0 kudos
- 752 Views
- 4 replies
- 1 kudos
Help regarding a python notebook and s3 file structure
Hello all, I am new to this forum, so please forgive if I am posting in the wrong location (I'd appreciate if the post is moved by mods or am told where to post).I am looking for help with an optimization of a python code I have. This python notebook...
- 752 Views
- 4 replies
- 1 kudos
- 1 kudos
I am not sure if I fully understand how your data pipeline is setup, but have you considered incremental data loading say using something similar to "COPY INTO" method which would only read your incremental load, and then apply a 90 day filter on top...
- 1 kudos
- 1348 Views
- 1 replies
- 0 kudos
Resolved! Generate embeddings for 50 million rows in dataframe
Hello All,I have dataframe with 5 million rows and before we can setup vector search endpoint against index, we want to generate embeddings column for each of those rows. Please suggest whats an optimal way to do this?We are in development phase so w...
- 1348 Views
- 1 replies
- 0 kudos
- 0 kudos
The easiest and most reliable way to generate embeddings for millions of rows is to let Databricks Vector Search compute them automatically during synchronization from a Delta table.Vector Search can generate embeddings for you, keep them updated whe...
- 0 kudos
- 425 Views
- 1 replies
- 0 kudos
Does Databricks Runtime 7.3+ include built-in Hadoop S3 connector configurations?
I came across the KB article S3 connection reset error, which mentions not using the following Spark settings for the Hadoop S3 connector for DBR 7.3 and above:spark.hadoop.fs.s3.impl com.databricks.s3a.S3AFileSystem spark.hadoop.fs.s3n.impl com.data...
- 425 Views
- 1 replies
- 0 kudos
- 0 kudos
No, you don’t need to set those on DBR 7.3 and above.From 7.3+ Databricks already uses the newer Hadoop S3A connector by default, so those com.databricks.s3a.S3AFileSystem settings are not part of the default config and shouldn’t be added.If they are...
- 0 kudos
- 2664 Views
- 2 replies
- 1 kudos
Output Not Displaying in Databricks Notebook on All-Purpose Compute Cluster
Hello All,I’m encountering an issue where output from standard Python commands such as print() or display(df) is not showing up correctly when running notebooks on an All-Purpose Compute cluster.Cluster DetailsCluster Type: All-Purpose ComputeRuntime...
- 2664 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi Surya, Do you face this issue only with DBR 17.3 all-purpose clusters? Did you try with lower DBRs? If not, please try and let me know. Also, from the Run menu, try “Clear state and outputs,” then re‑run the cell on the same cluster to rule out st...
- 1 kudos
- 4883 Views
- 2 replies
- 0 kudos
Can you default to `execution-count: none` when stripping notebook outputs?
When committing to a git folder, IPYNB outputs are usually stripped, unless allowed by an admin setting and toggled by .databricks/commit_outputs. This sets the{"execution-count": 0, ... }within the IPYNB metadata. Is there a way to set it instead to...
- 4883 Views
- 2 replies
- 0 kudos
- 0 kudos
Databricks does not currently allow you to default to "execution_count": null (or "none") when stripping notebook outputs during a commit. The platform sets "execution_count": 0 as the default when outputs are stripped through their Git integration, ...
- 0 kudos
- 4243 Views
- 2 replies
- 1 kudos
not able to install library in normal site while in community version it working please help
I am not able to install library in normal version while in community editioin i am able to add libray using compute how to install in normal databricks same as community edition.
- 4243 Views
- 2 replies
- 1 kudos
- 1 kudos
To install libraries in the normal (paid) version of Databricks, use the cluster management interface to add libraries to your compute resources. The process is similar to the Community Edition, but workspace policies and cluster access mode may rest...
- 1 kudos
- 3164 Views
- 1 replies
- 0 kudos
How to write ObjectId value using Spark connector 10.2.2
In pySpark mongo connector while updating records how to handle _id as objectId.spark 3.2.4scala2.13sparkMongoConnector 2.12-10.2.2
- 3164 Views
- 1 replies
- 0 kudos
- 0 kudos
To write an ObjectId value using Spark Mongo Connector 10.2.2 in PySpark while updating records, you must convert the ObjectId string into a special format. The Spark Mongo Connector does not automatically recognize a string as an ObjectId; it will o...
- 0 kudos
- 1241 Views
- 4 replies
- 0 kudos
Query to calculate cost of task from each job by day
I am trying to find the cost per Task in each Job every time it was executed (daily) but currently getting very huge numbers due to duplicates, can someone help me ? WITH workspace AS ( SELECT account_id, workspace_id, workspace_name,...
- 1241 Views
- 4 replies
- 0 kudos
- 0 kudos
You are seeing inflated cost numbers because your query groups by many columns—especially run_id, task_key, usage_start_time, and usage_end_time—without addressing possible duplicate row entries that arise from your joins, especially with the system....
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
