- 36145 Views
- 4 replies
- 1 kudos
Resolved! Is it possible to import functions from a module in Workspace/Shared instead of Repos?
Hi,I am considering creating libraries for my databricks notebooks, and found that it is possible to import functions from modules saved in repos. Is it possible to move the .py files with the functions to Workspace/Shared and still import functions ...
- 36145 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Christine Pedersen Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell ...
- 1 kudos
- 3429 Views
- 1 replies
- 2 kudos
Window function + Multiple simultaneous aggregations
Hi team,Why is there no support to perform multiple aggregations together with a single window spec? ie I dont want to specify each aggregation separately and I don't want to see each aggregation perform as a separate piece of work.Or if there is ind...
- 3429 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Kaniz Fatma ,Firstly, thank you much for responding.Thank you for confirming that performing multiple aggr using a single window spec does NOT evaluate the window spec separately each time. My bad in the wrong understanding prior.
- 2 kudos
- 1924 Views
- 0 replies
- 1 kudos
dbricks.co
Hi everyone!We’re excited to gather everyone for Data + AI Summit 2023 — the premier AI and LLM global event for the data, analytics and AI community. Join thousands of data engineers, data scientists and data analyst experts virtually from June 29–3...
- 1924 Views
- 0 replies
- 1 kudos
- 8293 Views
- 3 replies
- 0 kudos
I want to perform interpolation on a streaming table in delta live tables.
I have the following code:from pyspark.sql.functions import * !pip install dbl-tempo from tempo import TSDF from pyspark.sql.functions import * # interpolate target_cols column linearly for tsdf dataframe def interpolate_tsdf(tsdf_data, target_c...
- 8293 Views
- 3 replies
- 0 kudos
- 0 kudos
The issue was not resolved because we were trying to use a streaming table within TSDF which does not work.
- 0 kudos
- 24792 Views
- 8 replies
- 2 kudos
Resolved! AnalysisException when running SQL queries
When running some SQL queries using spark.sql(...), we sometimes get a variant of the following error:AnalysisException: Undefined function: current_timestamp. This function is neither a built-in/temporary function, nor a persistent function that is ...
- 24792 Views
- 8 replies
- 2 kudos
- 2 kudos
This is most likely a conflict in the lib code, you can uninstall some libs on your cluster and try to narrow it down to the problematic one.
- 2 kudos
- 2904 Views
- 2 replies
- 2 kudos
Resolved! Reduce downtime of Postgres table - JDBC overwrite job
I want to overwrite a Postgresql table transactionStats which is used by the customer facing dashboards.This table needs to be updated every 30 mins. I am writing a AWS Glue Spark job via JDBC connection to perform this operation.Spark dataframe writ...
- 2904 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Siddharth Kanojiya We haven't heard from you since the last response from @werners (Customer) . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
- 2 kudos
- 13526 Views
- 2 replies
- 2 kudos
Resolved! AZURE_QUOTA_EXCEEDED_EXCEPTION - even with more than vCPUs than Databricks recommends
I am running this Delta Live Tables PoC from databricks-industry-solutions/industry-solutions-blueprintshttps://github.com/databricks-industry-solutions/pos-dltI have Standard_DS4_v2 with 28GB and 8 cores x 2 workers - so a total of 16 cores. This is...
- 13526 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Prasenjit Biswas We haven't heard from you since the last response from @Jose Gonzalez . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
- 2 kudos
- 1688 Views
- 1 replies
- 2 kudos
This site has alot of issues....
I need to login and I can't to try Databricks..so you have a OAUTH issue...I cant try Databricks at all because the country icon doesn't work and it sends a URI issue from your front end to the back end..."Request_URI=&Geo_country_code=&Geo_country_i...
- 1688 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Jessica Woods Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 2 kudos
- 12150 Views
- 7 replies
- 3 kudos
Detectron2 install
I have been struggling to install Detectron 2. I think it is an issue with Cuda. Any adivise?install!pip install -q "detectron2@git+https://github.com/facebookresearch/detectron2.git@e2ce8dc#egg=detectron2"Error:qm8vsdal/detectron2_85faeed5ce7945dbad...
- 12150 Views
- 7 replies
- 3 kudos
- 3 kudos
Hi @Joshua RobergeSorry for the inconvenience!Kindly review the solution offered by @Suteja Kanuri.
- 3 kudos
- 32823 Views
- 21 replies
- 21 kudos
Resolved! How to access a jar file stored in Databricks Workspace ?
Hi All,We have a couple of jars stored in a workspace folder.We are using init scripts to copy the jars in the workspace to the /databricks/jars path.The init scripts do not seem to be able to find the files. The scripts are failing saying the files ...
- 32823 Views
- 21 replies
- 21 kudos
- 21 kudos
Hi @Bharath Kumar Ramachandran You're welcome! I'm glad you found the link useful. I empathize with your hope that Databricks would consider adding this option. It's possible that Databricks will take user feedback into account when planning future ...
- 21 kudos
- 3925 Views
- 4 replies
- 5 kudos
To do Optimization on the real time delta table
Hi Team,We have few prod tables which are created in s3 bucket, that have grown now very large, these tables are getting real time data continuously from round the clock databricks workflows; we would like run the optimization commands(optimize, zord...
- 3925 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi @Sriram Kumar We haven't heard from you since the last response from @Suteja Kanuri . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
- 5 kudos
- 15759 Views
- 7 replies
- 9 kudos
virtual environment on azure databricks compute cluster
I'm using Azure Databricks and I'd like to create a project virtual environment, persisted on a shared compute cluster. As the cluster is shared for many projects, it is necessary to have virtual environments if I want to execute code runs from withi...
- 15759 Views
- 7 replies
- 9 kudos
- 9 kudos
Hi @Joshua L We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ot...
- 9 kudos
- 1986 Views
- 1 replies
- 3 kudos
How to execute requests later for a number of times that exceeds the Maximum concurrent runs?
I am trying to start the same Jobs multiple times using the python sdk's "run_now" command.If the number of requests exceeds the Maximum concurrent runs, the status of the run will be Skipped and the run will not be executed.Is there any way to queue...
- 1986 Views
- 1 replies
- 3 kudos
- 3 kudos
Hi, We do have a private preview feature which will be enabled shortly for queueing. Please tag me (@Debayan Mukherjee ) with your next update so that I will get notified.
- 3 kudos
- 3815 Views
- 3 replies
- 3 kudos
Databricks run-time 12.2 LTS drop function problem
I am getting an error about the `drop function of pyspark` at a cluster using 12.2 LTS. When I check the error I see spark solved that bug, see SPARK-42444. Also when I check maintenance updates page, I saw this solved issue included the Databricks R...
- 3815 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Sevval Mehder Elevate our community by acknowledging exceptional contributions. Your participation in marking the best answers is a testament to our collective pursuit of knowledge.
- 3 kudos
- 2668 Views
- 2 replies
- 3 kudos
Resolved! Implement & Test DR Plan
Hi,Can you direct me to any documentation on how to implement and test Disaster Recovery for Databricks (PAAS) on Azure ?Thx & RgdsRamdas
- 2668 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Ramdas Panicher Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
