cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Ajay-Pandey
by Databricks MVP
  • 17143 Views
  • 7 replies
  • 11 kudos

Resolved! Unzip Files

Hi all, I am trying to unzip a file in databricks but facing an issue,Please help me if you have any doc or codes to share.

  • 17143 Views
  • 7 replies
  • 11 kudos
Latest Reply
vivek_rawat
New Contributor III
  • 11 kudos

Hey ajay,You can follow this module to unzip your zip file.To give your brief idea about this, it will unzip your file directly into your driver node storage.So If your compressed data is inside DBFS then you first have to move that to drive node and...

  • 11 kudos
6 More Replies
Firedubdub
by New Contributor II
  • 5179 Views
  • 2 replies
  • 4 kudos

Resolved! Azure DataBricks REST API 2.0 - Deactivate User by ID. I can't seem to find any more details on this anymore.

Hello team,I am looking to run Azure Logic Apps to deactivate user by ID. Right now there is only the REST API for delete user but I only want to deactivate the user.Can anyone help with the operations op/path/schema that I am missing?  Thank you!

image image.png
  • 5179 Views
  • 2 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Daren Liew​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 4 kudos
1 More Replies
PragyaS
by New Contributor
  • 3844 Views
  • 2 replies
  • 1 kudos

Resolved! Decimal Automatic Rounding on sum

I am facing an issue, I have implemented a code in which I am performing sum on decimal values. I have set precision as Decimal(19,2). When I am calculating for big data I am getting different value as I got from my .Net Utility application. ex. From...

  • 3844 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Pragya Sharma​ We haven't heard from you since the last response from @Werner Stinckens​  . Kindly share the information with us, and in return, we will provide you with the necessary solution. Thanks and Regards

  • 1 kudos
1 More Replies
lasribeiro_univ
by New Contributor III
  • 10754 Views
  • 3 replies
  • 3 kudos

Resolved! Cluster SQL Warehouse fails to Start.

I'm a new user of Databricks and I'm taking the Academy course, but I'm having difficulty starting a SQL Warehouse cluster. I've tried several different configurations, but I always get the same error:Clusters are failing to launch. Cluster launch wi...

SQL Warehouse Starter 2 SQL Warehouse Starter 1
  • 10754 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Luiz Ribeiro​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
2 More Replies
Manimeghala
by New Contributor III
  • 7277 Views
  • 4 replies
  • 5 kudos

Resolved! Can we use %run or any magic commands in a Delta Live Table pipeline?

When we try, it says magic commands are not supported. Is there a way to import other notebooks first so that its functions can be referenced/used while we build a DLT pipeline?

  • 7277 Views
  • 4 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Manimeghala Vemula​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...

  • 5 kudos
3 More Replies
Sudhir1
by New Contributor II
  • 10171 Views
  • 5 replies
  • 1 kudos

Connecting to AWS MSK

how to connect to the AWS MSK which has I-Am based authentication?

  • 10171 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Sudhir Jaiswal​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 1 kudos
4 More Replies
f2008700
by New Contributor III
  • 21642 Views
  • 6 replies
  • 7 kudos

Configuring average parquet file size

I have S3 as a data source containing sample TPC dataset (10G, 100G).I want to convert that into parquet files with an average size of about ~256MiB. What configuration parameter can I use to set that?I also need the data to be partitioned. And withi...

  • 21642 Views
  • 6 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @Vikas Goel​ We haven't heard from you since the last response from @Werner Stinckens​ â€‹, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...

  • 7 kudos
5 More Replies
adrianhernandez
by Databricks Partner
  • 13170 Views
  • 3 replies
  • 1 kudos

Resolved! Databricks Academy login issues

I tried using the same credentials as my Databricks login (provided by my employer) and it doesn't work. I then tried the forgot option password but never get an email. I finally tried registering a new account, but I get an error showing the email i...

  • 13170 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Adrian Hernandez​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...

  • 1 kudos
2 More Replies
Anonymous
by Not applicable
  • 7555 Views
  • 2 replies
  • 1 kudos
  • 7555 Views
  • 2 replies
  • 1 kudos
Latest Reply
wmespi
New Contributor II
  • 1 kudos

Is this random number not possible to extract from the notebook context? It is available in the browser_hash but that is not populated when running a job.Is this random number static or does it change over time? If it is static, it can then be hardco...

  • 1 kudos
1 More Replies
Mado
by Valued Contributor II
  • 9184 Views
  • 1 replies
  • 0 kudos

Resolved! Error when query a table created by DLT pipeline; "Couldn't find value of a column"

Hi, I create a table using DLT pipeline (triggered once). In the ETL process, I add a new column to the table with Null values by:output = output.withColumn('Indicator_Latest_Value_Date', F.lit(None))Pipeline works and I don't get any error. But, whe...

  • 9184 Views
  • 1 replies
  • 0 kudos
Latest Reply
josruiz22
New Contributor III
  • 0 kudos

Hi,Try converting the None of the output line this :output = output.withColumn('Indicator_Latest_Value_Date', F.lit(None).cast("String"))

  • 0 kudos
Ancil
by Contributor II
  • 7277 Views
  • 5 replies
  • 3 kudos

I have created one databricks account using my organisation email Id, i need to delete that account. Any one please help me how can we do that

I have created one databricks account using my organisation emailID, i need to delete that account. Any please help me how can we do that

  • 7277 Views
  • 5 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Ancil P A​ Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!

  • 3 kudos
4 More Replies
EDDatabricks
by Databricks Partner
  • 5249 Views
  • 3 replies
  • 7 kudos

Resolved! Unable to perform VACUUM on Delta table

We have a table containing records from the last 2-3 years. The table size is around 7.5 TBytes (67 Billion rows).Because there are periodic updates on historical records and daily optimizations of this table, we have tried repeatedly to execute a m...

  • 5249 Views
  • 3 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @EDDatabricks EDDatabricks​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that ...

  • 7 kudos
2 More Replies
carlosst01
by Databricks Partner
  • 3332 Views
  • 2 replies
  • 2 kudos

Resolved! Running Libraries and/or modules in Databricks' lifecycle?

Hi, i have had this question for some weeks and didn't find any information about the topic. Specifically, my doubt is: what is the 'lifecycle' or cycle or steps to be able to use a new Python library in Databricks in terms of compatibility? For exam...

  • 3332 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Carlos Caravantes​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 2 kudos
1 More Replies
adrianna2942842
by New Contributor III
  • 34399 Views
  • 3 replies
  • 7 kudos

Resolved! Exception "java.nio.charset.MalformedInputException: Input length = 1" when creating data profile on Docker Container Service (10.4 LTS)

I am encountering an issue while attempting to create a data profile on clusters using Docker Container Service (version 10.4 LTS). I keep receiving the following exception:java.nio.charset.MalformedInputException: Input length = 1What's puzzling is ...

  • 34399 Views
  • 3 replies
  • 7 kudos
Latest Reply
Vartika
Databricks Employee
  • 7 kudos

Hi @Adrianna Klank​,We haven't heard from you since the last response from @Akash Bhat​​, and I was checking back to see if the suggestion helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to other...

  • 7 kudos
2 More Replies
AhSon
by New Contributor II
  • 3431 Views
  • 2 replies
  • 5 kudos

Resolved! Databricks Certificate Renewal

I received email reminder on my databricks certificate that going to expire next month. May I check where we can renew the certificate like how we did in Microsoft?Thank you.

  • 3431 Views
  • 2 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Jason Yap​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...

  • 5 kudos
1 More Replies
Labels