- 3918 Views
- 2 replies
- 0 kudos
What is the best practice for tracing databricks - observe and writestream data record flow
Trying to connect dots on method below through a new event on Azure eventhub, storage, partition, avro records (those I can monitor) to my delta table? How do I trace observe, writeStream and the trigger? ... elif TABLE_TYPE == "live": print("D...
- 3918 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @David Martin​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 26533 Views
- 4 replies
- 1 kudos
Databricks cluster starts with docker
Hi there!I hope u are doing wellI'm trying to start a cluster with a docker image to install all the libraries that I have to use.I have the following Dockerfile to install only python libraries as you can seeFROM databricksruntime/standard WORKDIR /...
- 26533 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi! I am facing a similar issue.I tried to use this oneFROM databricksruntime/standard:10.4-LTS ENV DEBIAN_FRONTEND=noninteractive RUN apt update && apt install -y maven && rm -rf /var/lib/apt/lists/* RUN /databricks/python3/bin/pip install datab...
- 1 kudos
- 2012 Views
- 1 replies
- 21 kudos
Databricks Workflows is introducing a new feature called "Task Values", a simple API for setting and retrieving small values from tasks. Tas...
Databricks Workflows is introducing a new feature called "Task Values", a simple API for setting and retrieving small values from tasks. Tasks can now output values that can be referenced in subsequent tasks, making it easier to create more expressiv...
- 2012 Views
- 1 replies
- 21 kudos
- 21 kudos
We urgently hope for this feature, but to date, we have found that it is only available in Python. Do you have any plans to support Scala?Please provide more support for Scala.
- 21 kudos
- 3110 Views
- 2 replies
- 0 kudos
DBX injected V-Net and Deployment
Due to the need for Azure storage private endpoints, we switched our databricks deployment to use an injected vnet. Now, when our deployment pipeline tries to re-create the workspace (e.g. az databricks workspace delete), it seems to leave the MS cre...
- 3110 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @David Benedict​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 0 kudos
- 10761 Views
- 5 replies
- 1 kudos
Resolved! Autoloader directory listing not listing all files
Hi communityI have an Autoloader pipeline running with following configuration. Unfortunately, it does not detect all files. (see below query definition). The folder that needs to be read has 38.246 files that all have the same schema and structure.:...
- 10761 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @Fabrice Deseyn​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 1 kudos
- 4589 Views
- 6 replies
- 0 kudos
i am not able to reset the password for data bricks cloud community
Hi,i am not log in to https://community.cloud.databricks.com/login.html. Please assist .
- 4589 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Nilofar Sharma​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 0 kudos
- 8939 Views
- 3 replies
- 2 kudos
Resolved! Are window functions more performant than self joins?
I have a table with data for each month end and want to know the LEAD and LAG data points either side of each month. For example:SELECT month_date, LEAD(month_date) OVER (PARTITION BY id ORDER BY month_date) next_month_date, LAG(month_date) OVER (PA...
- 8939 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Matthew Elsham​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...
- 2 kudos
- 5655 Views
- 5 replies
- 3 kudos
Give customized notebook run id/name in console using dbutils.notebook.run()
I am running below code in Azure databricks:Now the running status shows as highlighted in yellow. I want to give a custom name to this running status instead of Notebook job #<job id>. This will help me identify which status is for which notebook so...
- 5655 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Gaurav Tanwar​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest...
- 3 kudos
- 3640 Views
- 2 replies
- 3 kudos
Resolved! Not able to reset password with password re-set link
I am not able to reset password https://community.cloud.databricks.com/When I use link provided in the email to re-set the password, and I enter new password, request never gets completed.Password reset link keeps on loading even after entering passw...
- 3640 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Shama Khadpekar​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedb...
- 3 kudos
- 4125 Views
- 3 replies
- 2 kudos
Bamboolib with Databricks
Hi Everyone,I am wondering if anyone has experience using the bamboolib library within Databricks. I am currently using it for a client to display table data on the UI and potentially allow users to edit existing rows and insert new ones. While I hav...
- 4125 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Chhaya Vishwakarma​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I sug...
- 2 kudos
- 47946 Views
- 4 replies
- 4 kudos
How to extract DDL from tables/views using a SQL query?
I want to write a SQL query that queries the information_schema to generate a list of objects, their columns, relationships etc. - basically a data dictionary. For each object I want to show the DDL code, and I know I can get it by executing show c...
- 47946 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Richard Architect​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I sugg...
- 4 kudos
- 4703 Views
- 2 replies
- 2 kudos
Resolved! Delta sharing (databricks-to-databricks) between azure regions issue
we have 2 unity meta stores in 2 regions ( useast2 contains data and westeurope contains clusters) and enabled delta sharing between them. We use azure storage firewall / vent whitelisting to allow secure connection to storage from compute cluste...
- 4703 Views
- 2 replies
- 2 kudos
- 2 kudos
@prasad vaze​ :Delta sharing between Unity metastores in different regions can be achieved in several ways, depending on your specific requirements and constraints. One common approach is to use Azure Private Link to establish a private connection be...
- 2 kudos
- 8845 Views
- 3 replies
- 1 kudos
dbsqlcli.main ERROR - getaddrinfo failed
I had to rebuild my laptop and am trying to re-install Databricks and Databricks-sql-cli so I can use the CLI to pull info from the databricks repository.When I try to run the command: dbsqlcli -e "select distinct tableName from databases_info.all_db...
- 8845 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @John Zajic​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1 kudos
- 5023 Views
- 2 replies
- 1 kudos
Resolved! Pyspark read multiple Parquet type expansion failure
ProblemReading nearly equivalent parquet tables in a directory with some having column X with type float and some with type double fails.Attempts at resolvingUsing streaming filesRemoving delta caching, vectorizationUsing ,cache() explicitlyNotesThis...
- 5023 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Erik Louie​ Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!Regards
- 1 kudos
- 8001 Views
- 6 replies
- 4 kudos
Databricks Attribute Error: 'IPythonShell' object has no attribute 'kernel'
I have been getting this error repeatedly when trying to run a notebook. I have tried attaching multiple different clusters and installing some of the libraries that it wanted me to update. I have also tried to clear the state of the notebook and res...
- 8001 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi @Toshali Mohapatra​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 4 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
