Apply Now! Are you passionate about data and want to make a difference for thousands of data practitioners? Databricks is looking for a dedicated and knowledgeable Community Technical Moderator to guide our thriving online community and empower users...
- 93 Views
- 0 replies
- 1 kudos
Wednesday
Meet Filip Niziol, a Senior Data Engineer at Digiterre and an active contributor to the Databricks Community. With a rich background in data architecture and a passion for building scalable data solutions, Filip shares insights on his career, Databri...
- 427 Views
- 4 replies
- 5 kudos
Monday
Each month, we recognize one outstanding member of our Databricks Community who has made a real impact. Whether you share solutions, spark discussions, or help others, this is your chance to get featured! What’s a Monthly Community Champion? Our ...
- 290 Views
- 0 replies
- 7 kudos
Monday
Over the years, organizations have amassed a vast amount of unstructured text data—documents, reports, and emails—but extracting meaningful insights has remained a challenge. Large Language Models (LLMs) now offer a scalable way to analyze this data,...
- 584 Views
- 0 replies
- 1 kudos
2 weeks ago
Migrating your data warehouse workloads is one of the most challenging yet essential tasks for any organization. Whether the motivation is the growth of your business and scalability requirements or reducing the high license and hardware cost of your...
- 1356 Views
- 0 replies
- 2 kudos
2 weeks ago
Community Activity
- 17 Views
- 1 replies
- 0 kudos
How to decide on creating views vs Tables in Gold layer?
We have the following use case:We receive raw form of data from an application and that is ingested in the Iron Layer. The raw data is in the JSON FormatThe Bronze layer will the first level of transformation. The flattening of the JSON file happens ...
- 17 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi Dharinip. I've had similar conversations internally with developers asking very similar things. This is my general advice for this situation, but I think there are a lot of considerations to how you create your gold layer.How large is this datas...
- 0 kudos
- 16 Views
- 1 replies
- 0 kudos
Notebooks running long in workflow
I have deployed a new databricks environment for development. I've copied a workflow from production to this environment with exactly the same compute configuration. Four notebooks that complete within minutes do not complete after 2 hours in develop...
- 16 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @trentlglover, It sounds like you're experiencing a significant performance issue with your notebooks in the new development environment. Here are a few potential areas to investigate: Cluster Configuration: Even though you mentioned that the comp...
- 0 kudos
- 16 Views
- 0 replies
- 0 kudos
Next user group meetup! December 10th.
Hello Phoenix Friends! Our final user group meeting for 2024 (where did the time go!) will be Tuesday December 10th. Register here. In addition to food, beverages, networking, and connecting, we have a couple of great speakers. We are thrilled to ha...
- 16 Views
- 0 replies
- 0 kudos
- 14 Views
- 0 replies
- 0 kudos
Executing Python code inside a SQL Function
Hi ,I am trying to create a SQL UDF and I am trying to run some python code involving pyspark, I am not able to create a spark session inside the python section of the function, here is how my code looks, CREATE OR REPLACE FUNCTION test.getValuesFro...
- 14 Views
- 0 replies
- 0 kudos
- 7 Views
- 0 replies
- 0 kudos
Salesforce LakeFlow connect - Deletion Salesforce records
Hello, I am new in databricks and related to data engineering. I am running a POC to sync data between a Salesforce sandbox and Databricks using LakeFlow connect.I already make the connection and i successfully sync data between salesforce and databr...
- 7 Views
- 0 replies
- 0 kudos
- 227 Views
- 7 replies
- 0 kudos
why latestOffset and getBatch takes so long time
Hi team,Kinesis -> delta table raw -> job with trigger=availableNow -> delta table target. The Kinesis->delta table raw is running continuously. The job is daily with trigger=availableNow. The job reads from raw, do some transformation, and run a MER...
- 227 Views
- 7 replies
- 0 kudos
Latest Reply
- 0 kudos
Thanks. I tracked there with log but cannot figure out which parts make the 18000 version apply slow. It is the same with CDF if I feed a big range to table_changes function. Any idea on this?
- 0 kudos
- 11 Views
- 0 replies
- 0 kudos
Idempotency token
Hi Team, I have used idempotency token in my dag code to avoid duplicate runs.note: Idempotency token given as static valueIssue: If dag fails once ...because of this idempotency token, airflow is not allowing to connect dbx ...can you please help me...
- 11 Views
- 0 replies
- 0 kudos
- 8855 Views
- 7 replies
- 3 kudos
Resolved! Download event, driver, and executor logs
Hi Team, I can see logs in Databricks console by navigating workflow -> job name -> logs. These logs are very generic like stdout, stderr and log4-avtive.log. How to download event, driver, and executor logs at once for a job? Regards,Rajesh.
- 8855 Views
- 7 replies
- 3 kudos
Latest Reply
- 3 kudos
@Kaniz Fatma​ @John Lourdu​ @Vidula Khanna​ Hi Team,I managed to download logs using the Databricks command line as below: Installed the Databricks command line on my Desktop (pip install databricks-cli)Configured the Databricks cluster URL and perso...
- 3 kudos
- 197 Views
- 5 replies
- 0 kudos
We are trying to connect to AWS RDS MySQL instance from DBX with PySpark using JDBC
We are trying to connect to AWS RDS MySQL instance from DBX with PySpark using JDBC: jdbc_df = (spark.read.format("jdbc").options(url=f"jdbc:mysql://{creds['host']}:{creds['port']}/{creds['database']}", driver="com.mysql.cj.jdbc.Driver", dbtable="(SE...
- 197 Views
- 5 replies
- 0 kudos
Latest Reply
- 0 kudos
Yes, that seems correct for the inbound traffic at least: Control plane services, including webapp: nvirginia.cloud.databricks.com, 3.237.73.224/28SCC relay: tunnel.us-east-1.cloud.databricks.comSCC relay for PrivateLink: tunnel.privatelink.us-east-1...
- 0 kudos
- 22 Views
- 0 replies
- 0 kudos
Not received my certificate after passing Data Engineer Associate exam
Hello, I took my exam and passed it successfully, but I still haven't received my certificate. You can help, because I need to deliver the certificate to my manager.My id is Gustavo.aps06@gmail.comtks
- 22 Views
- 0 replies
- 0 kudos
- 182 Views
- 3 replies
- 2 kudos
Serverless compute databricks
1. How to connect s3 bucket to databricks since dbfs mount is not supported.?2. In serverless compute Spark Context (sc), spark.sparkContext, and sqlContext are not supported?. Does it means it will not leverage power of distributed processing?3. Wha...
- 182 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
please see this documentation for accessing cloud storage by setting Unity Catalog objects: Storage Credential and External Location. https://docs.databricks.com/en/connect/unity-catalog/cloud-storage/index.html
- 2 kudos
- 56 Views
- 3 replies
- 0 kudos
Perform row_number() filter in autoloader
I have created an autoloader job that reads data from S3 (files with no extension) having json using (cloudFiles.format, text). Now this job is suppose to run every 4 hours and read all the new data that arrived. But before writing into a delta table...
- 56 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
HI @hkmodi ,Basically, as @daniel_sahal said, bronze layer should reflect the source system. The silver layer is dedicated for deduplication/cleaning/enrichment of dataset. If you still need to deduplicate at bronze layer you have 2 options:- use me...
- 0 kudos
- 6775 Views
- 4 replies
- 0 kudos
How to delete Databricks Account
I created and used a Databricks Account for testing purposes. I want to delete that account. In the Databricks Account Web UI, there is no menu to delete an account. How should I delete it?
- 6775 Views
- 4 replies
- 0 kudos
Latest Reply
- 0 kudos
I created and used a Databricks Account for testing purposes.I want to delete that account.In the Databricks Account Web UI, there is no menu to delete an account.How should I delete it?
- 0 kudos
- 537 Views
- 2 replies
- 0 kudos
Databricks JDBC driver fails with socket read timeout
We work with a application that connects to our Databricks serverless SQL warehouse via Databricks JDBC driver. It runs a few thousand SQL select statements everyday, and a small percentage of them will fail with the following error details: java.sql...
- 537 Views
- 2 replies
- 0 kudos
- 37 Views
- 0 replies
- 0 kudos
Temu code 100$ off
The Temu code for 100$ off is acu570611 code
- 37 Views
- 0 replies
- 0 kudos
Announcements
Latest from our Blog
Data Migration Decoded - Part 2: Creating and Running Tests
Echoing what we said in Part 1: Test Data Curation, when your team is migrating script code and pipeline code to Databricks, there are three main steps: Look at the original code, understand what it's...
Enterprise-wide data governance integrations with Unity Catalog
Unity Catalog is the foundation of all data governance-related aspects of the Databricks Data and Intelligence platform. It provides a unified, centralized solution for key enterprise capabilities, in...
How-to guide: Automating Media Metadata Classification with Databricks Workflows and LLMs
Have you ever sunk into your sofa after a long day, craving some TV time but faced with too many options? With hundreds of TV channels and several streaming services, you're overwhelmed. Programmers,...
Getting started with fine-tuning on Databricks
Authors: Ellen Hirt, Narjes Majdoub, Giran Moodley Contents ContentsIntroductionSection 1: Preparing the EnvironmentSection 2: Step-by-Step Instructions1. Data preparation2. Model initialisation and t...
