cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

varad
by New Contributor III
  • 6001 Views
  • 6 replies
  • 8 kudos

Resolved! My exam has suspended , Need help Urgently (10/06/2023)

Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam. I want to file a compla...

  • 6001 Views
  • 6 replies
  • 8 kudos
Latest Reply
Anonymous
Not applicable
  • 8 kudos

Hi @Varad Manglekar​ Glad to hear!Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!

  • 8 kudos
5 More Replies
mikimiki309
by New Contributor II
  • 4010 Views
  • 4 replies
  • 2 kudos

Resolved! Voucher not received

Hi, I have attended the recent Lakehouse webinar (May) and completed Lakehouse fundamentals but still have not received the certification voucher. Kindly help.

  • 4010 Views
  • 4 replies
  • 2 kudos
Latest Reply
mikimiki309
New Contributor II
  • 2 kudos

Thanks @Vidula Khanna​ for your help. I have raised a ticket as suggested for this.

  • 2 kudos
3 More Replies
naga_databricks
by Contributor
  • 16544 Views
  • 8 replies
  • 6 kudos

Resolved! Set timestamp column to blank when inserting a record into delta table

I am trying to insert a record into Delta table using notebook written in python. This record has a timestamp column that should be blank initially, later i have a plan to update the timestamp value.How am i inserting the record: stmt_insert_audit_r...

  • 16544 Views
  • 8 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Naga Vaibhav Elluru​ Elevate our community by acknowledging exceptional contributions. Your participation in marking the best answers is a testament to our collective pursuit of knowledge

  • 6 kudos
7 More Replies
Christine
by Contributor II
  • 35314 Views
  • 4 replies
  • 1 kudos

Resolved! Is it possible to import functions from a module in Workspace/Shared instead of Repos?

Hi,I am considering creating libraries for my databricks notebooks, and found that it is possible to import functions from modules saved in repos. Is it possible to move the .py files with the functions to Workspace/Shared and still import functions ...

  • 35314 Views
  • 4 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Christine Pedersen​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell ...

  • 1 kudos
3 More Replies
KarthikeyanB
by New Contributor II
  • 3272 Views
  • 1 replies
  • 2 kudos

Window function + Multiple simultaneous aggregations

Hi team,Why is there no support to perform multiple aggregations together with a single window spec? ie I dont want to specify each aggregation separately and I don't want to see each aggregation perform as a separate piece of work.Or if there is ind...

  • 3272 Views
  • 1 replies
  • 2 kudos
Latest Reply
KarthikeyanB
New Contributor II
  • 2 kudos

Hi @Kaniz Fatma​ ,Firstly, thank you much for responding.Thank you for confirming that performing multiple aggr using a single window spec does NOT evaluate the window spec separately each time. My bad in the wrong understanding prior.

  • 2 kudos
leelingmin
by Databricks Employee
  • 1820 Views
  • 0 replies
  • 1 kudos

dbricks.co

Hi everyone!We’re excited to gather everyone for Data + AI Summit 2023 — the premier AI and LLM global event for the data, analytics and AI community. Join thousands of data engineers, data scientists and data analyst experts virtually from June 29–3...

  • 1820 Views
  • 0 replies
  • 1 kudos
Eelke
by New Contributor II
  • 8067 Views
  • 3 replies
  • 0 kudos

I want to perform interpolation on a streaming table in delta live tables.

I have the following code:from pyspark.sql.functions import * !pip install dbl-tempo from tempo import TSDF   from pyspark.sql.functions import *   # interpolate target_cols column linearly for tsdf dataframe def interpolate_tsdf(tsdf_data, target_c...

  • 8067 Views
  • 3 replies
  • 0 kudos
Latest Reply
Eelke
New Contributor II
  • 0 kudos

The issue was not resolved because we were trying to use a streaming table within TSDF which does not work.

  • 0 kudos
2 More Replies
Merchiv
by New Contributor III
  • 23494 Views
  • 8 replies
  • 2 kudos

Resolved! AnalysisException when running SQL queries

When running some SQL queries using spark.sql(...), we sometimes get a variant of the following error:AnalysisException: Undefined function: current_timestamp. This function is neither a built-in/temporary function, nor a persistent function that is ...

  • 23494 Views
  • 8 replies
  • 2 kudos
Latest Reply
ashish1
Databricks Employee
  • 2 kudos

This is most likely a conflict in the lib code, you can uninstall some libs on your cluster and try to narrow it down to the problematic one.

  • 2 kudos
7 More Replies
siddharthk
by New Contributor II
  • 2699 Views
  • 2 replies
  • 2 kudos

Resolved! Reduce downtime of Postgres table - JDBC overwrite job

I want to overwrite a Postgresql table transactionStats which is used by the customer facing dashboards.This table needs to be updated every 30 mins. I am writing a AWS Glue Spark job via JDBC connection to perform this operation.Spark dataframe writ...

  • 2699 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Siddharth Kanojiya​ We haven't heard from you since the last response from @werners (Customer)​ . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards

  • 2 kudos
1 More Replies
Pras1
by New Contributor II
  • 12482 Views
  • 2 replies
  • 2 kudos

Resolved! AZURE_QUOTA_EXCEEDED_EXCEPTION - even with more than vCPUs than Databricks recommends

I am running this Delta Live Tables PoC from databricks-industry-solutions/industry-solutions-blueprintshttps://github.com/databricks-industry-solutions/pos-dltI have Standard_DS4_v2 with 28GB and 8 cores x 2 workers - so a total of 16 cores. This is...

  • 12482 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Prasenjit Biswas​ We haven't heard from you since the last response from @Jose Gonzalez​ â€‹ . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards

  • 2 kudos
1 More Replies
Woody
by New Contributor II
  • 1577 Views
  • 1 replies
  • 2 kudos

This site has alot of issues....

I need to login and I can't to try Databricks..so you have a OAUTH issue...I cant try Databricks at all because the country icon doesn't work and it sends a URI issue from your front end to the back end..."Request_URI=&Geo_country_code=&Geo_country_i...

  • 1577 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Jessica Woods​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...

  • 2 kudos
Westinghouse
by New Contributor II
  • 10140 Views
  • 7 replies
  • 3 kudos

Detectron2 install

I have been struggling to install Detectron 2. I think it is an issue with Cuda. Any adivise?install!pip install -q "detectron2@git+https://github.com/facebookresearch/detectron2.git@e2ce8dc#egg=detectron2"Error:qm8vsdal/detectron2_85faeed5ce7945dbad...

  • 10140 Views
  • 7 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Joshua Roberge​Sorry for the inconvenience!Kindly review the solution offered by @Suteja Kanuri.

  • 3 kudos
6 More Replies
ranged_coop
by Valued Contributor II
  • 31305 Views
  • 21 replies
  • 21 kudos

Resolved! How to access a jar file stored in Databricks Workspace ?

Hi All,We have a couple of jars stored in a workspace folder.We are using init scripts to copy the jars in the workspace to the /databricks/jars path.The init scripts do not seem to be able to find the files. The scripts are failing saying the files ...

  • 31305 Views
  • 21 replies
  • 21 kudos
Latest Reply
Anonymous
Not applicable
  • 21 kudos

Hi @Bharath Kumar Ramachandran​ You're welcome! I'm glad you found the link useful. I empathize with your hope that Databricks would consider adding this option. It's possible that Databricks will take user feedback into account when planning future ...

  • 21 kudos
20 More Replies
sriram_kumar
by New Contributor II
  • 3656 Views
  • 4 replies
  • 5 kudos

To do Optimization on the real time delta table

Hi Team,We have few prod tables which are created in s3 bucket, that have grown now very large, these tables are getting real time data continuously from round the clock databricks workflows; we would like run the optimization commands(optimize, zord...

  • 3656 Views
  • 4 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Sriram Kumar​ We haven't heard from you since the last response from @Suteja Kanuri​ â€‹ . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards

  • 5 kudos
3 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels