cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

17780
by New Contributor II
  • 23937 Views
  • 6 replies
  • 0 kudos

How to delete Databricks Account

I created and used a Databricks Account for testing purposes. I want to delete that account. In the Databricks Account Web UI, there is no menu to delete an account. How should I delete it?

  • 23937 Views
  • 6 replies
  • 0 kudos
Latest Reply
MadhuB
Contributor III
  • 0 kudos

Hi @17780 The easiest way is to delete the workspace and cancel your subscription.

  • 0 kudos
5 More Replies
Kuke
by New Contributor
  • 105 Views
  • 1 replies
  • 0 kudos

Missing Rows When Reading Data from Impala Kudu to Databricks Using JDBC

Hi everyone,I’m working on a data ingestion process where I need to read data from an Impala Kudu table into Databricks using the JDBC connector. However, I’m experiencing an issue where some rows are missing in the data read. For instance, if there ...

  • 105 Views
  • 1 replies
  • 0 kudos
Latest Reply
Takuya_Omi
Valued Contributor II
  • 0 kudos

@Kuke Have you checked whether the partitioning is configured correctly?If disabling partitioning (creating a single partition) allows you to retrieve 100,000 rows, but enabling partitioning results in only 99,000 rows, it is likely that the partitio...

  • 0 kudos
dunno
by New Contributor II
  • 551 Views
  • 5 replies
  • 0 kudos

Resolved! How to Dynamically Retrieve Serverless Cluster ID for Databricks Job Configuration?

I am working on deploying a Databricks job to the production environment using a PowerShell script in Azure DevOps release pipeline. The task requires to update the job configuration JSON file to set the job's compute to serverless. For this, I need ...

  • 551 Views
  • 5 replies
  • 0 kudos
Latest Reply
Walter_C
Databricks Employee
  • 0 kudos

sure, happy to assist, if any of my responses was able to help you would really appreciate if you can accept it as a solution 

  • 0 kudos
4 More Replies
Andrewcon
by New Contributor II
  • 2175 Views
  • 2 replies
  • 1 kudos

Delta tables and YOLO computer vision tasks

 Hi all,I would really appreciate if someone could help me out. I feel it’s both a data engineering and ML question.One thing we use at wo is YOLO for object detection. I’ve managed to run YOLO by loading data from the blob storage, but I’ve seen tha...

Data Engineering
computer vision
Delta table
YOLO
  • 2175 Views
  • 2 replies
  • 1 kudos
Latest Reply
MathieuDB
Databricks Employee
  • 1 kudos

Hello @Andrewcon and @jnap , Have a look at Mosaic Streaming Dataset. You could load your data from your delta table and then train it on your PyTorch YOLO model. In that example, it use mobilenet model but you can adapt it to use YOLO. Petastorm is ...

  • 1 kudos
1 More Replies
maxutil
by New Contributor II
  • 17562 Views
  • 6 replies
  • 3 kudos

Resolved! Invalid Characters in Column Names " ,;{}()\n\t="

I'm reading data into a dataframe withdf = spark.read.json("s3://somepath/")I've tried first creating a delta table using the DeltaTable API with:DeltaTable.createIfNotExists(spark)\ .location(target_path)\ .addColumns(df.sche...

  • 17562 Views
  • 6 replies
  • 3 kudos
Latest Reply
VZLA
Databricks Employee
  • 3 kudos

Glad it helped @jb1z , happy to help.

  • 3 kudos
5 More Replies
User16776430979
by New Contributor III
  • 51007 Views
  • 4 replies
  • 5 kudos

Best practices around bronze/silver/gold (medallion model) data lake classification?

What's the best way to organize our data lake and delta setup? We’re trying to use the bronze, silver and gold classification strategy. The main question is how do we know what classification the data is inside Databricks if there’s no actual physica...

  • 51007 Views
  • 4 replies
  • 5 kudos
Latest Reply
G_E
New Contributor II
  • 5 kudos

Has the reply from @Retired_mod been removed?

  • 5 kudos
3 More Replies
SQL
by New Contributor II
  • 2551 Views
  • 6 replies
  • 1 kudos

Presto hive table to delta table conversion

Hi Everyone, I am using the below sql query to generate the days in order in hive & it is working fine. The table got migrated to delta and my query is failing. It would be appreciated if someone helps me to figure out the issue.SQL Query :with  ex...

  • 2551 Views
  • 6 replies
  • 1 kudos
Latest Reply
thelogicplus
Contributor
  • 1 kudos

Hi @SQL @jose_gonzalez , Have you tried code conversion tool fromTravinto technologies  ? They have hive to delta table conversion 

  • 1 kudos
5 More Replies
subhankar
by New Contributor II
  • 493 Views
  • 2 replies
  • 0 kudos

Need guidance on connecting to Azure Databricks using JDBC Protocol

Step 1: Download and Reference the JDBC Driver Download the Databricks JDBC Driver: Visit the Databricks JDBC Driver download page. Download the appropriate version for your operating system. Extract the DatabricksJDBC42.jar file from the downloaded ...

  • 493 Views
  • 2 replies
  • 0 kudos
Latest Reply
Ayushi_Suthar
Databricks Employee
  • 0 kudos

Hi @subhankar , Good Day!  Looking at the error you are getting here shows that it tries to find some kind of JVM file and probably refers to the JAVA_HOME variable to achieve it. It looks as if it is not set correctly in your Environment Variables. ...

  • 0 kudos
1 More Replies
aliacovella
by Contributor
  • 298 Views
  • 3 replies
  • 1 kudos

How can I get logging or print output from a Delta Live Table workflow.

I'm trying to debug a task that is a DLT workflow and I've tried putting in log statements and print statements but I can't seem to see the output in the event log after the run nor can I see the print statements anywhere. Can someone point me to whe...

  • 298 Views
  • 3 replies
  • 1 kudos
Latest Reply
Edthehead
Contributor III
  • 1 kudos

Refer to this answer https://community.databricks.com/t5/data-engineering/how-to-print-out-logs-during-dlt-pipeline-run/td-p/82303 

  • 1 kudos
2 More Replies
Sangeetha112
by New Contributor
  • 1625 Views
  • 1 replies
  • 0 kudos

Email Extraction

Hi , Hope you are doing well. I was trying to extract a specific email attachment from the outlook, and inject into the dbfs loaction, but something went wrong. Could you please help. I am hereby giving the code whcih I used.  import imaplibimport em...

  • 1625 Views
  • 1 replies
  • 0 kudos
Latest Reply
Stefan-Koch
Valued Contributor
  • 0 kudos

If you face issues with IMAP, consider using Microsoft Graph API for email access. It provides robust support for Outlook without handling IMAP details and enhances security with OAuth2 tokens.Followed is a sample script, but I didn't tested it: pip ...

  • 0 kudos
DeepankarB
by New Contributor III
  • 801 Views
  • 2 replies
  • 1 kudos

Resolved! Error API calling with Service Principal Secret

Hi,I am working on Databricks workspace setup on AWS and trying to use Service Principal to execute API calls (CI/CD) deployment through Bitbucket. So I created secret for the service principal and trying to test the token. The test failed with below...

  • 801 Views
  • 2 replies
  • 1 kudos
Latest Reply
DeepankarB
New Contributor III
  • 1 kudos

I have been able to resolve this issue. Apparently you need to generate access token using service principal client id and client secret.  saurabh18cs solution is more relevant to Azure Databricks. Got below link from Databricks which provide generic...

  • 1 kudos
1 More Replies
gourishrivastav
by New Contributor
  • 511 Views
  • 1 replies
  • 0 kudos

Resolved! Databricks Fundamentals Certificate

Dear Team,I have successfully completed the Databricks Fundamentals training and aced the certificate quiz with a perfect score of 200 out of 200. However, I have not yet received the certificate. Can you please let me know the expected timeline for ...

  • 511 Views
  • 1 replies
  • 0 kudos
Latest Reply
Satyadeepak
Databricks Employee
  • 0 kudos

You should be able to receive it immediately. Can you share your use id with which you have taken the quiz?

  • 0 kudos
krocodl
by Contributor
  • 8917 Views
  • 12 replies
  • 3 kudos

OOM while loading a lot of data through JDBC

   public void bigDataTest() throws Exception { int rowsCount = 100_000; int colSize = 1024; int colCount = 12; String colValue = "'"+"x".repeat(colSize)+"'"; String query = "select explode(s...

Screenshot 2023-10-13 at 08.10.08.png Screenshot 2023-10-13 at 08.12.52.png
Data Engineering
JDBC
Out-of-memory
resource leaking
  • 8917 Views
  • 12 replies
  • 3 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 3 kudos

@Retired_modany idea?

  • 3 kudos
11 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels