- 4482 Views
- 4 replies
- 1 kudos
How to read feature table without target_df / online inference based on filter_condition in databricks feature store
I'm using databricks feature store == 0.6.1. After I register my feature table with `create_feature_table` and write data with `write_Table` I want to read that feature_table based on filter conditions ( may be on time stamp column ) without calling ...
- 4482 Views
- 4 replies
- 1 kudos
- 1 kudos
create_training_set is just a simple Select from delta tables. All feature tables are just registered delta tables. Here is an example code that I used to handle that: customer_features_df = spark.sql("SELECT * FROM recommender_system.customer_fea...
- 1 kudos
- 7165 Views
- 2 replies
- 1 kudos
Resolved! How to use SQL UDFs for Delta Live Table pipelines?
I've been searching for a way to use a SQL UDF for our DLT pipeline. In this case it is to convert a time duration string into INT seconds. How exactly do we use/apply UDFs in this case?
- 7165 Views
- 2 replies
- 1 kudos
- 1 kudos
@GimYou can create Python UDF and then use it in SQL.https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-cookbook.html#use-python-udfs-in-sql
- 1 kudos
- 4063 Views
- 0 replies
- 1 kudos
How to pass all dag_run.conf parameters to python_wheel_task
I want to trigger Databricks job from Airflow using DatabricksSubmitRunDeferrableOperator and I need to pass configuration params. Here is excerpt from my code (definition is not complete, only crucial properties):from airflow.providers.databricks.op...
- 4063 Views
- 0 replies
- 1 kudos
- 5433 Views
- 3 replies
- 2 kudos
Resolved! Unable to reproduce Kmeans Clustering results even after setting seed and tolerance
Hi I have been trying to reproduce Kmeans results with no luckHere is my code snippet:from pyspark.ml.clustering import KMeansKMeans(featuresCol=featuresCol, k=clusters, maxIter=40, seed=1, tol = .00001) Can anyone help?
- 5433 Views
- 3 replies
- 2 kudos
- 2 kudos
This issue was due to spark parallelization which doesn't guarantee the same data is assigned to each partition. I was able to resolve this by making sure the same data is assigned to the same partitions :df.repartition(num_partitions, "ur_col_id")d...
- 2 kudos
- 7149 Views
- 3 replies
- 4 kudos
Resolved! Does Databricks offer something like Oracle's dblink?
I am aware, I can load anything into a DataFrame using JDBC, that works well from Oracle sources. Is there an equivalent in Spark SQL, so I can combine datasets as well?Basically something like so - you get the idea...select lt.field1, rt.fie...
- 7149 Views
- 3 replies
- 4 kudos
- 12814 Views
- 5 replies
- 10 kudos
Resolved! Databricks clusters stuck on Pending and Terminating state indefinitely
Hi everyone,Our company is using Databricks on GKE. It works fine until suddenly when we try to create and terminate clusters today, it got stuck on Pending and Terminating state for hours (now more than 6 hours). There is no conclusion can be drawn ...
- 12814 Views
- 5 replies
- 10 kudos
- 10 kudos
Hi @Kurnianto Trilaksono Sutjipto​ : Figured out after multiple connects that This is typically a cloud provider issue. You can file a support ticket if the issue persists.
- 10 kudos
- 14043 Views
- 2 replies
- 3 kudos
Resolved! Trigger on a table
Hello! Is there an equivalent of Create trigger on a table in Databricks sql?CREATE TRIGGER [schema_name.]trigger_nameON table_nameAFTER {[INSERT],[UPDATE],[DELETE]}[NOT FOR REPLICATION]AS{sql_statements}Thank you in advance!
- 14043 Views
- 2 replies
- 3 kudos
- 3 kudos
You can try Auto Loader: Auto Loader supports two modes for detecting new files: directory listing and file notification.Directory listing: Auto Loader identifies new files by listing the input directory. Directory listing mode allows you to quickly ...
- 3 kudos
- 1486 Views
- 1 replies
- 0 kudos
Fail to load excel data(timeout) in databricks sample notebook
Im working with the sample notebook named '1_Customer Lifetimes.py' in https://github.com/databricks-industry-solutions/customer-lifetime-valueIn notebook, there is the code like this `%run "./config/Data Extract"`This load excel data however it occu...
- 1486 Views
- 1 replies
- 0 kudos
- 0 kudos
@Seungsu Lee​ It could be a destination host issue, configuration issue or network issue.Hard to guess, first check if your cluster has an access to the public internet by running this command:%sh ping -c 2 google.com
- 0 kudos
- 5094 Views
- 1 replies
- 0 kudos
Parent Hierarchy Queries/ Path Function /Recursive CTE's
Problem Statement:We have a scenario where we get the data from the source in the format of (in actual 20 Levels and number of fields are more than 4 but for ease of understanding let’s consider below)The actual code involved 20 levels of 4-5 fields ...
- 5094 Views
- 1 replies
- 0 kudos
- 0 kudos
I don't think that we have anything similar as a built-in function. You'll need to write some custom code to achieve that.
- 0 kudos
- 9878 Views
- 11 replies
- 13 kudos
Resolved! Is it possible to use other databases within Delta Live Tables (DLT)?
I have set up a DLT with "testing" set as the target database. I need to join data that exists in a "keys" table in my "beta" database, but I get an AccessDeniedException, despite having full access to both databases via a normal notebook.A snippet d...
- 9878 Views
- 11 replies
- 13 kudos
- 13 kudos
As an update to this issue: I was running the DLT pipeline on a personal cluster that had an instance profile defined (as per databricks best practises). As a result, the pipeline did not have permission to access other s3 resources (e.g other databa...
- 13 kudos
- 10394 Views
- 4 replies
- 3 kudos
Getting error while loading parquet data into Postgres (using spark-postgres library) ClassNotFoundException: Failed to find data source: postgres. Please find packages at http://spark.apache.org/third-party-projects.html Caused by: ClassNotFoundException
Hi Fellas - I'm trying to load parquet data (in GCS location) into Postgres DB (google cloud) . For bulk upload data into PG we are using (spark-postgres library)https://framagit.org/interhop/library/spark-etl/-/tree/master/spark-postgres/src/main/sc...
- 10394 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Kaniz Fatma​ , @Daniel Sahal​ - Few updates from my side.After so many hits and trials , psycopg2 worked out in my case.We can process 200+GB data with 10 node cluster (n2-highmem-4,32 GB Memory, 4 Cores) and driver 32 GB Memory, 4 Cores with Run...
- 3 kudos
- 1617 Views
- 1 replies
- 0 kudos
How are parallel and subsequent jobs handled by cluster?
Hello,Apologize for dumb question but i'm new to Databricks and need clarification on following.Are parallel and subsequent jobs able to reuse the same compute resources to keep time and cost overhead as low as possible vs. are they spinning a new cl...
- 1617 Views
- 1 replies
- 0 kudos
- 0 kudos
@tanja.savic tanja.savic​ You can use shared job cluster:https://docs.databricks.com/workflows/jobs/jobs.html#use-shared-job-clustersBut remember that a shared job cluster is scoped to a single job run, and cannot be used by other jobs or runs of the...
- 0 kudos
- 4821 Views
- 3 replies
- 1 kudos
Resolved! PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output was 1 and the length of input was 2.'.
I have pandas_udf, its working for 1 rows, but I tried with more than one rows getting below error.PythonException: 'RuntimeError: The length of output in Scalar iterator pandas UDF should be the same with the input's; however, the length of output w...
- 4821 Views
- 3 replies
- 1 kudos
- 1 kudos
I was testing, and your function is correct. So you need to have an error in inputData type (is all string) or with result_json. Please also check the runtime version. I was using 11 LTS.
- 1 kudos
- 7147 Views
- 5 replies
- 3 kudos
Resolved! Exporting R data frame variable
Hi all.I am trying to export R data frame variable as csv file.I am using this formula:df<- data.frame(VALIDADOR_FIM)df.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").save("dbfs:/FileStore/df/df.csv")But isn´t working. ...
- 7147 Views
- 5 replies
- 3 kudos
- 3 kudos
Please try to execute write.csv with the following path instead:write.csv(TotalData,file='/dbfs/tmp/df.csv',row.names = FALSE)%fs ls /tmp
- 3 kudos
- 27190 Views
- 21 replies
- 11 kudos
java.lang.IllegalArgumentException: java.net.URISyntaxException
I am using Databricks Autoloader to load JSON files from ADLS gen2 incrementally in directory listing mode. All source filename has Timestamp on them. The autoloader works perfectly couple of days with the below configuration and breaks the next day ...
- 27190 Views
- 21 replies
- 11 kudos
- 11 kudos
Hi Everyone,I'm seeing this issue as well - same configuration of the previous posts, using autoloader with incremental file listing turned on. The strange part is that it mostly works despite almost all of the files we're loading having colons incl...
- 11 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
