- 7812 Views
- 7 replies
- 7 kudos
Resolved! Spark eventlog for Cluster pools
Hi,I would want to setup the cluster logging (to capture eventlogs to /dbfs/cluster-logs dir) in my cluster pool configuration? is it possible?If I create cluster manually, I could able to setup the cluster logging as mentioned here: https://docs.mic...
- 7812 Views
- 7 replies
- 7 kudos
- 7 kudos
Hi @Stephanraj C​ instance pool is to reduce cluster start and auto-scaling times for a cluster. Are you using any API to create clusters? If so could you please share the API request?
- 7 kudos
- 4161 Views
- 2 replies
- 5 kudos
Error while mounting ADLS Gen 2 storage account to Az Databricks
Hello TeamI am facing troubles while mounting storage account onto my databricks. Some background on my setup:Storage Account - stgAcc1 - attached to vnet1 and it's subnetsDatabricks - databricks1 - attached to 'workers-vnet' and subnets - these were...
- 4161 Views
- 2 replies
- 5 kudos
- 5 kudos
Hi @Pranith Gaddam​ Does @Debayan Mukherjee​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 5 kudos
- 6030 Views
- 2 replies
- 1 kudos
Salesforce to Databricks
Hello Team,I am trying to run the salesforce and try to extract the data.AT that time i am facing the below issue :SOURCE_SYSTEM_NAME = 'Salesforce'TABLE_NAME = 'XY'desc = eval("sf." + TABLE_NAME + ".describe()")print(desc)for field in desc['fields']...
- 6030 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Rohit Kulkarni​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 1 kudos
- 3560 Views
- 2 replies
- 0 kudos
Is there any way we can use usermetadataAsOf option in time travelling query or can we modify the timestamps of delta lake that seems to be immutable?
We are using delta lake time travelling capability in our current project. We can use select * from timestamp/versionAsOF query. However ,there might be some change in our approach and we might need to recreate the delta lake while persisting the tim...
- 3560 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi Priyanka, Thanks for reaching out to community.databricks.com. As of now, "As of" query only has two parameters timestamp and version. Please refer: https://docs.delta.io/latest/delta-batch.html#sql-as-of-syntaxPlease let us know in case if you ha...
- 0 kudos
- 3796 Views
- 5 replies
- 5 kudos
Not able to install geomesa on my Databricks cluster
Hi team, I have attempting to install Geomesa (2.12:3.4.1) library on my cluster but it keeps failing with the below error:Library installation attempted on the driver node of cluster 0824-052900-76icyj32 and failed. Please refer to the following err...
- 3796 Views
- 5 replies
- 5 kudos
- 5 kudos
Hi @Ayushi Pandey​ I could see the package is available in the maven repo. https://mvnrepository.com/artifact/org.locationtech.geomesa/geomesa_2.12/3.4.1Have you tried downloading the package to dbfs location and installed on the cluster?
- 5 kudos
- 2171 Views
- 1 replies
- 2 kudos
Why am I getting a FileNotFoundError after providing the file path?
I used copy file path to get the file path of the notebook I am trying to run from another notebook.file_path = "/Users/ankur.lohiya@workday.com/PAS/Training/Ingest/TrainingQueries-Cloned.py/"ddi = DatabricksDataIngestion(file_path=file_path, ...
- 2171 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Ankur Lohiya​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...
- 2 kudos
- 4528 Views
- 2 replies
- 1 kudos
Driver context not found for python spark for spark_submit_task using Jobs API submit run endpoint
I am trying to run a multi file python job in databricks without using notebooks. I have tried setting this up by:creating a docker image using the DBRT 10.4 LTS as a base and adding the zipped python application to that.make a call to the run submit...
- 4528 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Umar Khan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 1 kudos
- 4473 Views
- 2 replies
- 1 kudos
Resolved! What are the options to offer a low latency API for small tables derived from big tables?
I have a big dataset which gets divided into smaller datasets. For some of these smaller datasets I'd like to offer a low latency API (*** ms) to query them. Big dataset 1B entriesSmaller dataset 1 Mio entriesWhat's the best way to do it?I thought ab...
- 4473 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Jan R​ Does @Tian Tan​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 3882 Views
- 2 replies
- 0 kudos
Resolved! Writing transformed DataFrame to a persistent table is unbearable slow
I want to transform a DF with a simple UDF. Afterwards I want to store the resulting DF in a new table (see code below)key = "test_key" schema = StructType([ StructField("***", StringType(), True), StructField("yyy", StringType(), True), StructF...
- 3882 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Jan R​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 0 kudos
- 1786 Views
- 2 replies
- 1 kudos
I need help finding the right mode for my course
How do I find the Data Brick Community edition?
- 1786 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Kester Truman​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 1 kudos
- 4175 Views
- 2 replies
- 2 kudos
Create dropdown-list in Markdown
In the first cell of my notebooks, I record a changelog for all changes done in the notebook in Markdown. However, as this list becomes longer and longer, I want to implement a dropdown list. Is there anyway to do this in Markdown in databricks?For t...
- 4175 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Jesse vd S​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 2 kudos
- 1580 Views
- 1 replies
- 0 kudos
A basic DataFrame transformation query
I want to know how dataframe transformations work.Suppose I have a DataFrame instance df1. I apply some operation on it, say a filter. As every operation gives a new dataframe, so lets say now we have df2. So we have two DataFrame instances now, df1 ...
- 1580 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @mghildiy​ Does @Kaniz Fatma​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 0 kudos
- 2832 Views
- 2 replies
- 2 kudos
Resolved! Where is Databricks Tunnel (and is Databricks connect cool again?)
Two related questions:1: There has been several mentions in this forum about "Databricks Tunnel", which should allow us to connect from our local IDE to a remote databricks cluster and develop stuff locally. The roumors said early 2022, is there some...
- 2832 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi there @Erik Parmann​ Does @Youssef Mrini​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks
- 2 kudos
- 2242 Views
- 3 replies
- 1 kudos
Any plans to provide Databricks SQL / Alerts API
Hi, Databricks! You are my favorite Big Data tool, but I've recently faced an issue I didn't expect to have. For our agriculture customers, we're trying to use Databricks SQL Platform to keep our data accurate all day. We use Alerts to validate our d...
- 2242 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Dmytro Imshenetskyi​ Does @Hubert Dudek​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 1 kudos
- 1559 Views
- 1 replies
- 0 kudos
Cannot sign in at databricks partner-academy portal
Hi thereI have used my company email to register an account for customer-academy.databricks.com a while back.Now what I need to do is create an account with partner-academy.databricks.com using my company email too.However when I register at partner-...
- 1559 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
