cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

ankris
by New Contributor III
  • 14149 Views
  • 5 replies
  • 6 kudos

Resolved! Databricks and Web app connectivity to build a interactive application

Hi All,Could you please guide and provide me some material on "Databricks and Web app connectivity to build a interactive application"Objective is to build web app so that user can interact for getting different data/visuals from databricks (pyspark ...

  • 14149 Views
  • 5 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @ananthakrishna raikar​ We haven't heard from you since the last response from @Werner Stinckens​ â€‹ . Kindly share the information with us, and in return, we will provide you with the necessary solution. Thanks and Regards

  • 6 kudos
4 More Replies
Ajay-Pandey
by Esteemed Contributor III
  • 10944 Views
  • 7 replies
  • 11 kudos

Resolved! Unzip Files

Hi all, I am trying to unzip a file in databricks but facing an issue,Please help me if you have any doc or codes to share.

  • 10944 Views
  • 7 replies
  • 11 kudos
Latest Reply
vivek_rawat
New Contributor III
  • 11 kudos

Hey ajay,You can follow this module to unzip your zip file.To give your brief idea about this, it will unzip your file directly into your driver node storage.So If your compressed data is inside DBFS then you first have to move that to drive node and...

  • 11 kudos
6 More Replies
Firedubdub
by New Contributor II
  • 2966 Views
  • 2 replies
  • 4 kudos

Resolved! Azure DataBricks REST API 2.0 - Deactivate User by ID. I can't seem to find any more details on this anymore.

Hello team,I am looking to run Azure Logic Apps to deactivate user by ID. Right now there is only the REST API for delete user but I only want to deactivate the user.Can anyone help with the operations op/path/schema that I am missing?  Thank you!

image image.png
  • 2966 Views
  • 2 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Daren Liew​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 4 kudos
1 More Replies
PragyaS
by New Contributor
  • 2058 Views
  • 2 replies
  • 1 kudos

Resolved! Decimal Automatic Rounding on sum

I am facing an issue, I have implemented a code in which I am performing sum on decimal values. I have set precision as Decimal(19,2). When I am calculating for big data I am getting different value as I got from my .Net Utility application. ex. From...

  • 2058 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Pragya Sharma​ We haven't heard from you since the last response from @Werner Stinckens​  . Kindly share the information with us, and in return, we will provide you with the necessary solution. Thanks and Regards

  • 1 kudos
1 More Replies
lasribeiro_univ
by New Contributor III
  • 6147 Views
  • 3 replies
  • 3 kudos

Resolved! Cluster SQL Warehouse fails to Start.

I'm a new user of Databricks and I'm taking the Academy course, but I'm having difficulty starting a SQL Warehouse cluster. I've tried several different configurations, but I always get the same error:Clusters are failing to launch. Cluster launch wi...

SQL Warehouse Starter 2 SQL Warehouse Starter 1
  • 6147 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Luiz Ribeiro​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 3 kudos
2 More Replies
Manimeghala
by New Contributor III
  • 3575 Views
  • 4 replies
  • 5 kudos

Resolved! Can we use %run or any magic commands in a Delta Live Table pipeline?

When we try, it says magic commands are not supported. Is there a way to import other notebooks first so that its functions can be referenced/used while we build a DLT pipeline?

  • 3575 Views
  • 4 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Manimeghala Vemula​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...

  • 5 kudos
3 More Replies
Sudhir1
by New Contributor II
  • 6380 Views
  • 5 replies
  • 1 kudos

Connecting to AWS MSK

how to connect to the AWS MSK which has I-Am based authentication?

  • 6380 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Sudhir Jaiswal​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 1 kudos
4 More Replies
f2008700
by New Contributor III
  • 14534 Views
  • 6 replies
  • 7 kudos

Configuring average parquet file size

I have S3 as a data source containing sample TPC dataset (10G, 100G).I want to convert that into parquet files with an average size of about ~256MiB. What configuration parameter can I use to set that?I also need the data to be partitioned. And withi...

  • 14534 Views
  • 6 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @Vikas Goel​ We haven't heard from you since the last response from @Werner Stinckens​ â€‹, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to o...

  • 7 kudos
5 More Replies
adrianhernandez
by New Contributor III
  • 2568 Views
  • 3 replies
  • 1 kudos

Resolved! Databricks Academy login issues

I tried using the same credentials as my Databricks login (provided by my employer) and it doesn't work. I then tried the forgot option password but never get an email. I finally tried registering a new account, but I get an error showing the email i...

  • 2568 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Adrian Hernandez​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answ...

  • 1 kudos
2 More Replies
Anonymous
by Not applicable
  • 5260 Views
  • 2 replies
  • 1 kudos
  • 5260 Views
  • 2 replies
  • 1 kudos
Latest Reply
wmespi
New Contributor II
  • 1 kudos

Is this random number not possible to extract from the notebook context? It is available in the browser_hash but that is not populated when running a job.Is this random number static or does it change over time? If it is static, it can then be hardco...

  • 1 kudos
1 More Replies
Mado
by Valued Contributor II
  • 5080 Views
  • 1 replies
  • 0 kudos

Resolved! Error when query a table created by DLT pipeline; "Couldn't find value of a column"

Hi, I create a table using DLT pipeline (triggered once). In the ETL process, I add a new column to the table with Null values by:output = output.withColumn('Indicator_Latest_Value_Date', F.lit(None))Pipeline works and I don't get any error. But, whe...

  • 5080 Views
  • 1 replies
  • 0 kudos
Latest Reply
josruiz22
New Contributor III
  • 0 kudos

Hi,Try converting the None of the output line this :output = output.withColumn('Indicator_Latest_Value_Date', F.lit(None).cast("String"))

  • 0 kudos
Ancil
by Contributor II
  • 5060 Views
  • 5 replies
  • 3 kudos

I have created one databricks account using my organisation email Id, i need to delete that account. Any one please help me how can we do that

I have created one databricks account using my organisation emailID, i need to delete that account. Any please help me how can we do that

  • 5060 Views
  • 5 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hey there @Ancil P A​ Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!

  • 3 kudos
4 More Replies
EDDatabricks
by Contributor
  • 2970 Views
  • 3 replies
  • 7 kudos

Resolved! Unable to perform VACUUM on Delta table

We have a table containing records from the last 2-3 years. The table size is around 7.5 TBytes (67 Billion rows).Because there are periodic updates on historical records and daily optimizations of this table, we have tried repeatedly to execute a m...

  • 2970 Views
  • 3 replies
  • 7 kudos
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @EDDatabricks EDDatabricks​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that ...

  • 7 kudos
2 More Replies
carlosst01
by New Contributor II
  • 1827 Views
  • 2 replies
  • 2 kudos

Resolved! Running Libraries and/or modules in Databricks' lifecycle?

Hi, i have had this question for some weeks and didn't find any information about the topic. Specifically, my doubt is: what is the 'lifecycle' or cycle or steps to be able to use a new Python library in Databricks in terms of compatibility? For exam...

  • 1827 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Carlos Caravantes​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 2 kudos
1 More Replies
adrianna2942842
by New Contributor III
  • 21453 Views
  • 3 replies
  • 7 kudos

Resolved! Exception "java.nio.charset.MalformedInputException: Input length = 1" when creating data profile on Docker Container Service (10.4 LTS)

I am encountering an issue while attempting to create a data profile on clusters using Docker Container Service (version 10.4 LTS). I keep receiving the following exception:java.nio.charset.MalformedInputException: Input length = 1What's puzzling is ...

  • 21453 Views
  • 3 replies
  • 7 kudos
Latest Reply
Vartika
Databricks Employee
  • 7 kudos

Hi @Adrianna Klank​,We haven't heard from you since the last response from @Akash Bhat​​, and I was checking back to see if the suggestion helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to other...

  • 7 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels